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FOREWORD 


The  essays  in  this  technical  report  represent  presenta¬ 
tions  made  as  part  of  a  symposium  entitled,  "Selection  of 
Air  Traffic  Controllers:  Complexity,  Requirements,  and 
Public  Interest."  The  symposium  was  presented  at  the  98th 
Annual  Convention  of  the  American  Psychological  Asso¬ 
ciation,  August  10-14,  in  Boston,  MA.  The  presentations 
address  the  diverse  process  of  valid  selection  for  a  highly- 
demanding  occupation  in  the  Federal  Civil  Service:  the  Air 
Traffic  Control  Specialist. 

Although  the  military  services  and  a  growing  number 
of  colleges  and  universities  provide  training  in  air  traffic 
control,  the  Federal  Aviation  Administration  has  up  to  now 
maintained  fairly  strict  and  restricted  access  to  most  of  the 
required  selection  and  training  for  this  highly-visible  job 
whose  occupants  are  perceived  to  hold  the  nation's  air 
traffic  and  safety  in  their  hands. 

Each  of  the  accompanying  reports  discusses  a  differ¬ 
ent  component  of  the  selection  process  for  air  traffic 
controllers. 
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OVERVIEW  OF  NATIONAL  AIRSPACE 
HUMAN  RESOURCE  MANAGEMENT  PLANNING  PROCESS 


The  modernization  of  the  National  Airspace 
System  (NAS)  was  undertaken  by  the  Federal 
Aviation  Administration  (FA A),  industry,  and  the 
aviation  community  to  meet  increases  in  system 
capacity  demands.  Unprecedented  growth  in  the 
number  of  aircraft  operations  is  requiring  im¬ 
proved,  expanded  services,  additional  equipment, 
improved  work  force  productivity,  and  the 
orderly  replacement  of  aging  equipment. 

To  meet  these  demands,  FA  A  has  examined 
its  operational,  technical,  and  personnel  require¬ 
ments  and  designed  one  of  the  most  ambitious 
technological  and  organizational  transition  efforts 
f-ver  attempted.  Over  a  20-year  period,  new  tech¬ 
nological  advances  will  be  introduced  involving 
higher  levels  of  automation,  in  addition  to  major 
consolidation  efforts  which  will  transform  all  en 
route  centers  and  188  terminal  radar  approach 
control  facilities  into  fewer  than  30  major  air 
traffic  control  facilities. 

The  success  of  the  introduction  of  new  sys¬ 
tems  and  the  consolidation  of  facilities  are  depen¬ 
dent  upon  16,000  Air  Traffic  (AT)  controllers, 
8,700  Airway  Facilities  (AF)  systems  specialists, 
and  thousands  of  managers  and  support  personnel 
at  the  FAA  Academy,  Technical  Center,  and  the 
Logistics  Center.  To  transition  smoothly  from 
the  existing  air  traffic  control  system  to  a  mod¬ 
ernized  NAS  will  be  dependent  upon  FAA’s 
ability  to  develop  and  implement  a  long-range 
plan  for  managing  the  human  resource  aspects  of 
the  NAS  modernization.  Such  a  long-range  plan 
will  ensure  that  during  the  modernization  FAA 
has  adequate  numbers  of  properly  trained  people 
in  the  right  place  at  the  right  time. 

The  purposes  of  this  paper  are  to  (a)  describe 
the  context  in  which  human  resource  planning 
must  be  accomplished,  (b)  describe  the  planning 
approach/analyses  process  that  identifies  (NAS) 
human  resource  requirements  and  strategies  to 
address  them,  and  (c)  examine  strategies  to 
ensure  ownership  of  institutionalization  of  the 
planning  process. 

The  Planning  Context.  Figure  1  depicts 


the  complex  human  machine  system  that  we  call 
our  National  Airspace  System  (NAS).  Human 
resource  planning  has  to  be  accomplished  within 
the  context  of  the  systems  that  make  up  the  NAS 
and  the  overall  NAS  performance  requirements 

The  three  elements  contained  in  Figure  1  are 
highly  interactive,  and  as  any  two  of  the  elements 
become  "fixed"  the  requirements  and  constraints 
for  the  third  element  are  clarified,  ror  e.v -unple, 
in  planning  for  the  NAS  modernization,  one  of 
the  basic  assumptions  is  that  there  is  no  margin 
for  reduction  in  either  safety  or  capacity  dimen¬ 
sions  of  NAS  performance.  Given  these  fixed 
performance  requirements,  as  the  design  of  the 
systems  in  the  NAS  becomes  fixed,  the  perfor¬ 
mance  requirements  for  the  human  resource 
element  in  the  NAS  also  become  fixed.  For  this 
reason,  it  is  important  that  the  human  resource 
element  be  considered  as  early  as  possible  in  the 
design  of  systems.  Ideally,  human  resource  anal¬ 
yses  can  be  conducted  before  systems  designs  are 
"fixed"  so  that  potential  limitations  or  costs  of 
the  human  resource  component  can  be  considered 
in  systems  design. 

Even  under  conditions  of  fixed  systems 
design,  there  is  still  an  opportunity  to  enhance 
the  human  contribution  to  the  overall  system 
performance  by  developing  good  opera- 
tionai/maintenance  policies  and  procedures. 
Operational  policies  and  procedures  and  main¬ 
tenance  policies  and  procedures,  along  with 
software,  define  the  allocation  of  work  between 
the  systems  and  the  human.  A  second  oppor¬ 
tunity  to  enhance  the  contribution  of  the  human 
is  the  human  resource  pipeline  itself.  FAA  must 
ensure  that  adequate  numbers  of  properly  trained 
people  are  available. 

Within  the  described  context,  it  is  essential 
that  human  resource  planning  be  based  on 
analyses  that  incorporate  data  and  assumptions 
regarding  technology,  people,  and  systems  per¬ 
formance  requirements. 

The  Approach.  Figure  2  illustrates  the 
basic  approach  to  NAS  human  resource  planning 
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Figure  1.  NAS  Human  Resource  Planning  Context 
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that  FAA  began  in  1988.  The  approach  is  based 
upon  four  major  phases  with  associated  analytical 
processes. 

The  first  phase,  contextual  analysis,  identifies 
the  end  state(s)  or  the  planning  target  for  the 
human  resource  planning  process.  The  principal 
objectives  of  the  contextual  analyses  are  to  define 
the  future  work  environment  and  to  identify  the 
critical  human  resource  management  issues  that 
must  be  addressed  in  the  planning  process. 

The  principal  analysis  conducted  during  this 
phase  is  an  alternative  futures  analysis.  The 
objective  of  the  analysis  to  identify  critical  events 
and  trends  that  will  impact  the  human  resource 
(i.e.,  operational  concept  for  future  control  of  air 
traffic,  future  maintenance  concepts,  etc.).  The 
output  is  a  set  of  "alternative  futures"  scenarios 
defined  by  future  operations  and  maintenance 
concepts  as  well  as  future  systems  capabilities 
and  specifications. 

Once  the  planning  targets  have  been  iden¬ 
tified,  the  next  phase,  human  resource  analyses, 
is  conducted  to:  (a)  identify  the  human  resources 
required  to  support  the  projected  future  environ¬ 
ment;  (b)  project  the  capacity  of  the  human 
resource  pipeline  to  meet  the  projected  demand. 

In  identifying  the  human  resource  "demand", 
the  pertinent  analyses  focus  on  estimating  the 
basic  workload  for  air  traffic  controllers  and  AF 
systems  specialists  who  will  operate  and  maintain 
the  components  of  the  future  NAS.  The  resulting 
workload  estimates  form  the  basis  for  answering 
the  question  of  how  many  people  are  needed  to 
operate  the  new  systems. 

Attention  is  then  given  to  the  knowledge, 
skills,  and  abilities  (KSA’s)  that  are  required  to 
effectively  operate  and  maintain  the  future  NAS. 
Concurrently,  analyses  focus  on  the  viability  of 
different  or ^anizational  structures  as  a  function  of 
variables  such  a;  management  structure,  shift 
schedules,  location  of  human  resources,  etc. 

The  analyses  conducted  in  the  first  two  steps 
of  the  hu.r.a.i  resource  analyses  piia.>c  auunss  the 
operational  human  resource  requirements  for  the 
future  state.  A  third  set  of  analyses  are  conduc¬ 


ted  to  identify  the  resources  for  transition-specific 
related  activities  such  as  transition  training, 
operational  test  and  evaluation,  shadow  opera¬ 
tions  during  equipment  installation  and  shake- 
down. 

The  ability  of  FAA  to  meet  the  projected 
human  resource  requirements  is  analyzed  through 
sets  of  pipeline  analyses  focused  on  the  "supply" 
side.  Separate  analyses  are  conducted  to  examine 
the  recruitment,  selection,  and  training  systems 
for  various  work  forces.  In  addition  to  addres¬ 
sing  issues  such  as  selection  rates,  attrition  rates, 
and  lead  times  required  for  various  selection  and 
training  activities,  the  pipeline  analyses  should 
examine  the  potential  capacity  of  various  systems 
in  the  pipeline  to  meet  demands  and  requirements 
for  additional  resources  such  as  staffing. 

Additionally  during  this  phase,  analyses  focus 
on  the  training  resource  requirements  and  sched¬ 
uling  implications  associated  with  alternative 
training  strategies.  The  analyses  conducted  in  this 
portion  of  the  human  resource  planning  program 
do  not  address  content  and  media  selection  for 
specific  training  programs.  Instead,  the  analyses 
are  focused  on  issues  such  as  the  number  of 
personnel  required  for  training  in  a  given  period 
c\f  time,  timing,  and  length  of  training  under 
alternative  training  strategies.  Analyses  will  be 
conducted  to  provide  separate  and  combined 
resource  requirements  for  the  AT  and  AF  work 
forces. 

Once  the  planning  targets  of  the  future 
environment  have  been  defined  and  the  human 
resources  required  to  support  that  environment 
have  been  estimated,  the  FAA  then  must  be  able 
to  determine  how  it  will  meet  the  identified 
needs.  As  illustrated  in  Figure  2,  the  policy 
analysis  phase  of  the  approach  involves  a  three- 
step  iterative  process  that  eventually  produces 
outcomes  such  as  human  resource  policies, 
recommendations  for  human  resource  manage¬ 
ment  programs,  future  system  design  recommen¬ 
dations,  and  recommendations  for  operations  and 
maintenance  policies.  The  iterative  policy  analy¬ 
sis  process  begins  with  the  development  of 
alternative  human  resource  piograms  that  repre¬ 
sent  alternative  courses  of  action  to  resolve  issues 
identified  through  the  human  resource  analyses. 
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Figure  2.  Human  Resource  Planning  Approach 


The  alternative  programs  are  then  subjected  to 
impact  analyses  which  produce  estimates  of  the 
system  impacts  and  costs  associated  with  each 
human  resource  program  alternative.  Policy 
makers  then  examine  these  alternative  human 
resource  programs  and  their  associated  costs  and 
performance  impacts  within  the  context  of  other 
operational  policy  factors  known  to  the  policy 
makers.  These  individuals  may  suggest  revisions 
in  the  alternatives  which  are  then  subjected  to  the 
impact  analyses  and  examined  by  the  policy 
makers  during  the  next  iteration  of  the  policy 
analysis  process.  When  the  policy  makers  have 
examined  all  feasible  alternatives,  they  develop 
the  appropriate  recommendations  regarding  hu¬ 
man  resource  policy,  programs,  etc. 

The  final  phase  of  the  planning  approach 
illustrated  in  Figure  2  is  an  evaluation  phase. 
During  this  phase,  human  resource  policies  and 
programs,  as  well  as  system  design  recommenda¬ 
tions,  are  evaluated  for  feasibility,  practicality, 
and  cost.  This  evaluation  is  an  ongoing  process 
and  provides  feedback  to  allow  for  adjustments 
to  HR  programs  that  have  been  developed,  as 
well  as  system  performance  requirements,  sys¬ 
tems  design,  and  operations  and  maintenance 
concepts  that  have  been  established.  The  infor¬ 
mation  from  these  evaluation  efforts  is  then  fed 
back  into  the  planning  process  and  considered  as 
part  of  the  context  for  future  iterations  of  the 
planning  cycle. 

Institutionalizing  the  Planning  Process.  In 
1988  the  FAA  initiated  the  NAS  Human  Re¬ 
source  Management  program.  This  program’s 
primary  goal  is  to  support  the  FAA  in  develop¬ 
ing  an  effective,  efficient  process  for  managing 
the  human  resource  aspects  of  the  NAS  mod¬ 
ernization.  Through  use  of  the  planning  approach 
described  in  the  previous  section,  the  FAA  will 
be  able  to  achieve  this  goal  because  it  will  have 
a  mechanism  to  do  the  following: 

Develop  and  implement  an  integrated  strategy 
for  recruitment,  selection,  training  and  manage¬ 
ment  of  huiiiao  icoources; 

•  Provide  FAA  managers  with  accurate 
information  to  estimate  training  require¬ 


ments  and  evaluate  alternative  training  strate¬ 
gies; 

•  Provide  FAA  managers  with  data  required  to 
project  and  obtain  adequate  funding  to  imple¬ 
ment  the  required  human  resource  manage¬ 
ment  programs; 

•  Provide  managers  in  the  field  with  the  ap¬ 
propriate  data  and  planning  tools  to  develop 
and  implement  solutions  for  human  resource 
challenge; 

•  Provide  the  FAA  work  force  with  infor¬ 
mation  which  established  realistic  expec¬ 
tations  and  effective  means  to  cope  with 
changes  resulting  from  the  NAS  moderniza¬ 
tion;  and 

•  Establish  an  ongoing  human  resource  plan¬ 
ning  process  that  is  integrated  with  and 
sensitive  to  the  evolving  requirements  and 
changing  schedules  of  the  NAS  moderniza¬ 
tion  effort. 

The  management  of  such  a  comprehensive 
planning  program  in  an  agency  such  as  the  FAA 
represents  a  considerable  challenge  with  changing 
operational  requirements,  time  and  budget  con¬ 
straints.  The  requirements  of  the  program  and 
the  existing  FAA  management  culture  suggest  the 
need  for  a  management  process  that  is  highly 
participative  and  highly  integrated.  The  success 
of  such  a  management  process  is  highly  depen¬ 
dent  on  good  communication  and  buy  in  to  the 
planning  approach  by  all  major  stake  holders. 

Given  the  management  requirements  cited 
above,  the  FAA  developed  and  implemented  a 
management  process  for  the  NAS  HRM  Program 
that  operates  with  the  following  philosophy: 

•  The  most  immediate  and  essential  require¬ 
ments  for  human  resource  planning  exist  in 
the  operational  environment  of  the  FAA; 

•  Human  resource  planning  is  the  respon¬ 
sibility  of  FAA  line  management; 

•  The  office  of  the  Associate  Administrator  for 
Human  Resource  Management  (AHR)  can 


provide  methodological  expertise  tor  NAS 
human  resource  planning  and  develop  pro¬ 
grams  to  facilitate  change  in  the  FAA.  hut 
the  executive  directors  are  the  policy 
makers  and  policy  implemented;  and 

•  Ownership  of  the  NAS  Human  Resource 
Plan  must  reside  with  FAA  line  manage¬ 
ment. 

The  emphasis  on  line  management  respon¬ 
sibility  for  human  resource  planning  which  per¬ 
vades  the  assumptions  above  is  not  only  consis¬ 
tent  with  existing  FAA  culture,  but  also  has  been 
associated  with  successful  human  resource  pro¬ 
grams  in  other  organizations.  Such  a  manage¬ 
ment  philosophy  does,  however,  require  a  sig¬ 
nificant  investment  of  time  and  other  resources 
In  senior  managers  involved  in  FAA  operations 
and  system  development. 

To  assure  involvement  by  FAA  managers  in 
providing  direction  and  establishing  management 
policy  rm  the  continued  development  and  im¬ 
plementation  of  the  planning  process,  the  FAA 
Fxecutive  Board  chartered  two  management 
committees-the  NAS  HRM  Steering  Committee 
and  the  Working  Committee.  The  NAS  HRM 
Steering  Committee  is  tasked  with  making  deci¬ 
sions  regarding  policy  development  and  im¬ 
plementation.  The  NAS  HRM  Working  Commit¬ 
tee  is  tasked  with  providing  data,  subject-matter 
experts,  and  coordinating  planning  activities  for 
each  participating  line  organization. 

The  Office  of  Human  Resource  Development 
and  the  line  organizations  develop  a  Memoran¬ 
dum  of  Understanding  (MOU)  that  documents 
what  needs  to  be  accomplished  in  support  of  the 
planning  process.  The  MOU’s  give  detailed 
information  on  project  activities,  critical  mile¬ 
stones,  resources  required,  and  organizational 
points  of  contact. 

Status.  The  first  iteration  of  the  NAS  HRM 
Plan  will  be  published  in  October  6.  1 090 . 
Development  of  the  1 90 1  NAS  HRM  Plan  is 
underway. 


NOTE:  The  technical  support  for  the  devel¬ 
opment  of  the  1990  NAS  HRM  Plan  was  pro¬ 
vided  bv  FU  Associates,  Ltd.,  under  the  very 
able  direction  of  Dr.  Barry  Rieglehaupt. 
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EMPLOYING  AIR  TRAFFIC  CONTROLLERS 


The  process  of  recruiting,  selecting,  and 
placing  air  traffic  controllers  is  in  many  ways  a 
model  personnel  problem.  It  has  several  features 
that  make  it  similar  to  the  large  scale  military 
and  heavy  industry  selection  situations  from 
which  much  of  classic  personnel  testing  is  de¬ 
rive!  At  the  same  time,  it  has  other  features  that 
reflect  the  problems  we  all  face  in  personnel 
selection  today. 

I  will  describe  how  the  FA  A  hires  cont.  oilers, 
and  as  I  do,  I  will  point  out  the  kinds  of  prob¬ 
lems  and  parallels  that  I  just  mentioned. 

The  FAA  currently  employs  about  17,000  air 
traffic  controllers  in  terminal  facilities  and  en 
route  centers.  These  controllers  are  civil  ser¬ 
vants.  They  have  civil  service  appointments,  just 
like  IRS  agents.  Social  Security  clerks,  and  Park 
Rangers.  We  hire  between  1800  and  3400  new 
controllers  each  year  using  civil  service  proce¬ 
dures.  The  FAA  trains  all  these  people  as  con¬ 
trollers.  Thus,  we  engage  in  mass  entry  level 
hiring  and  training,  like  the  military  and  some 
industries.  We  have  to  use  procedures  that  can 
help  us  screen  and  s  "t  large  numbers  of  people 
for  the  ability  to  learn  the  job  and  perform  in  it. 

The  controllers’  strike  in  1981  resulted  in  the 
firing  of  over  half  of  the  controller  work  force. 
Because  of  the  importance  of  aviation  to  this 
country  and  public  concerns  about  safety,  our 
ability  to  rebuild  that  work  force  is  the  object  of 
intense  public  and  Congressional  scrutiny.  This 
constant  scrutiny  is  one  aspect  of  the  environ¬ 
ment  in  which  we  conduct  our  daily  business  of 
employing  controllers.  Partly  in  reaction  to  the 
pressure  to  rebuild  the  work  force  as  quickly  as 
possible,  the  FAA  has  become  a  leader  in  devel¬ 
oping  streamlined  civil  service  hiring  procedures. 

Recruitment.  We  begin  with  recruitment. 
We  recruit  in  a  number  of  ways.  Our  controllers 
recruit  their  sons,  daughters,  cousins,  nephews, 
nieces,  and  friends  They  also  make  presenta¬ 
tions  at  schools  and  meetings  of  community 
go -ups  Besides  the  controllers  themselves,  we 
have  a  growing  staff  of  full  time  recruiters  in  our 


human  resources  offices  who  go  to  schools, 
community  organizations,  job  fairs,  and  air 
shows  in  search  of  likely  candidates.  W'e  also 
advertise.  We  use  newspaper  ads,  magazine  ads, 
television  ads,  and  in  some  places  toll  free 
numbers  to  obtain  responses. 

We  must  recruit  constantly.  In  recent  years, 
the  controller  job  has  received  a  great  deal  of 
negative  publicity,  particularly  with  regard  to  the 
stress  in  the  job. 

More  than  that,  as  a  Federal  employer,  we 
have  an  affirmative  action  obligation  that  only 
recruitment  can  help  us  meet.  A  piece  of  that 
Congressional  scrutiny  I  mentioned  before  con¬ 
cerns  our  EEO  posture.  I  cannot  over  emphasize 
the  level  of  interest  by  Congress  and  minority 
groups  in  the  FAA’s  hiring  of  controllers  or  the 
seriousness  with  which  w'e  take  that  interest 

Finally,  aggressive  recruitment  is  necessary 
because  it  is  hard  for  us  to  find  young  people 
with  the  combination  of  abilities  needed  for 
controller  work.  By  civil  service  policy,  we  do 
not  hire  anyone  under  18  years  of  age,  except 
that  we  could  hire  a  16  year  old  who  had  gradu¬ 
ated  from  high  school.  We  also  have  a  maxi¬ 
mum  entry  age  for  terminal  and  en  route  posi¬ 
tions.  Applicants  may  not  have  reached  their  31st 
birthday  prior  to  initial  appointments  to  those 
positions  Our  preferred  applicants  are  in  their 
twenties.  That  younger  population  is  declining  in 
this  country.  The  baby  boom  is  long  over  for  us. 
We  are  competing  with  the  military,  other  public 
safety  jobs  that  have  maximum  entry  ages,  and 
business  for  an  increasingly  scarce  human  re¬ 
source.  This  is  a  real  problem  that  our  re  ruiters 
grapple  with  every  day. 

Age.  Let  me  point  out  that  the  maximum 
entry  age  is  not  popular  with  the  people  it  ex¬ 
cludes  from  employment.  A  good  portion  of  my 
time  as  a  policy  maker  is  spent  responding  to 
criticism  of  the  age  limitation  rule.  We  have 
numerous  studies  supporting  the  fact  that  people 
do  not  do  well  in  the  controller  occupation  as 
they  get  older.  This  holds  for  a  xyrietv  of  enter 
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ion  measures.  Moreover,  the  maximum  entry 
age  is  specifically  provided  for  by  law  and  the 
government  has  successfully  defended  the  rule 
against  legal  challenges. 

Along  with  the  maximum  entry  age,  con¬ 
trollers  have  a  mandatory  separation  age  of  56. 
By  law,  controllers  must  be  removed  from 
positions  requiring  direct  separation  and  control 
of  air  traffic  by  that  age.  The  law  is  actually 
somewhat  more  complex  than  I  have  described  it. 
It  also  entitles  controllers  to  full  retirement  much 
earlier  than  the  average  civil  servant.  They 
regard  this  as  a  significant  benefit.  The  effect  of 
'hose  rules  is  that  we  recruit  people  who  are 
young  and  they  leave  the  occupation  relatively 
young. 

Selection.  The  Federal  Civil  Service  is  a 
merit  system.  We  are  supposed  tv)  find  the  best 
available  people  for  johs.  For  this  reason,  air 
traffic  controllers  are  normally  hired  under 
competitive  procedures.  That  means  we  must 
have  an  examination  by  which  all  qualified 
applicants  can  compete  on  the  basis  of  their 
ability.  Applicants  compete  by  taking  a  written 
civil  service  test  battery  designed  to  select  con¬ 
trollers. 

The  government  usually  does  not  use  off- 
the-shelf  commercial  exams  for  civil  service 
tests,  partly  for  test  security  reasons.  The  tests 
in  the  battery  were  developed  by  the  Office  of 
Personnel  Management  (OPM)  and  the  FAA. 
Normally,  tnese  civil  service  tests  must  be  vali¬ 
dated  prior  to  implementation  as  operational 
exams.  This  was  the  case  with  our  exam.  Fur¬ 
ther,  civil  service  tests  are  subject  to  the  Uniform 
Guidelines  on  Employee  Selection  Procedures 
and  all  the  attendant  court  decisions,  interpreta¬ 
tions,  and  data  collection  requirements  imposed 
on  any  large  employer,  government  or  business, 
in  the  United  States. 

Our  current  test  consists  of  three  parts.  One 
part  is  composed  of  figure  classification  and 
letter  series  problems.  Another  part  consists  of 
drawings  which  simulate  a  radar  scope  depicting 
patterns  of  air  traffic.  A  table  containing  flight 
information  about  the  airciaft.  like  speed,  al¬ 
titude,  and  route  is  also  pr  waled.  The  cx 


aminees  answer  questions  that  use  the  informa¬ 
tion  provided  by  file  drawings  and  tables.  For 
example,  they  answer  questions  about  which 
aircraft  are  likely  to  conflict  with  each  other  and 
the  distances  between  aircraft.  This  part  of  the 
battery  does  not  require  that  the  applicant  know 
or  apply  air  traffic  control  knowledge.  Know¬ 
ledge  is  assessed  in  the  third  part  of  the  examina¬ 
tion.  The  knowledge  test  is  intended  to  give  extra 
credit  to  those  who  have  previous  air  traffic 
control  experience  or  extensive  aviation  ex¬ 
perience,  for  example,  people  who  have  been  air 
traffic  controllers  for  the  military. 

The  OPM  has  responsibility  for  civil  service 
testing.  Both  the  F/\A  and  OPM  administer  the 
test.  OPM  delegated  the  FAA  authority  to 
administer  the  test  to  help  us  reach  more  ap¬ 
plicants.  Together,  we  test  between  25,000  and 
100.000  people  a  year  We  try  to  offer  the  exam 
throughout  the  U.S.  If  we  make  people  travel 
long  distances  to  take  the  test,  we  are,  in  effect, 
discriminating  against  those  from  lower  socioeco¬ 
nomic  strata  who  cannot  afford  to  travel. 

The  test  is  sometimes  subject  to  criticism 
because  members  of  certain  minority  groups  and 
women  generally  do  not  score  as  high  on  the  test 
as  we  would  like.  This  is  not  an  unusual  situ¬ 
ation  for  an  employer.  In  addition,  as  with  some 
other  examinations  for  government  jobs  and  some 
college  entrance  examinations,  there  are  a  num¬ 
ber  of  companies  now  offering  books  and  semi¬ 
nars  on  how  to  take  the  test.  We  have  data 
showing  that  portions  of  the  battery  are  subject  to 
practice  effects  and  this  is  a  subject  of  concern. 

When  applicants  sit  for  the  test,  they  also  fill 
out  a  computer  form  that  lets  us  determine  if 
they  are  probably  eligible  for  employment  as 
controllers.  In  addition  to  the  restrictions  I 
spoke  of,  applicants  may  not  be  former  con¬ 
trollers  fired  for  striking,  they  must  be  U.  S. 
citizens,  and  have  three  years  of  general  work 
experience  of  any  type  or  four  years  of  college 
or  a  combination  of  the  two  equaling  three  yeais 
of  work  experience. 

All  Civil  Service  tests  must  he  scaled  for  a 
passing  score  of  70  and  a  maximum  of  100.  An 
exception  here  is  veteran’s  preference.  The 


8 


veteran’s  preference  rules  require  that  extra 
points  be  added  to  the  test  scores  of  certain 
veterans  in  order  to  produce  a  final  rating. 
Veteran’s  preference  points  can  take  a  rating  over 
the  100  mark.  A  score  of  75.1  on  the  test  is 
passing  for  those  with  no  prior  aviation  ex¬ 
perience.  Those  with  certain  kinds  of  aviation 
related  work  experience  or  certain  educational 
qualifications  need  only  score  70  on  the  test. 

We  rarely  consider  people  with  final  ratings 
in  the  70’s.  Those  are  low  scores.  Our  prefer¬ 
ence  is  to  hire  people  with  scores  of  90  or  above. 
When  sufficient  numbers  of  people  with  scores  in 
the  90’s  are  not  available,  we  will  consider 
people  with  scores  in  the  high  80’s. 

The  answer  sheets  are  scored  first  in  the  FAA 
office  responsible  for  testing  in  that  state.  This 
unofficial  initial  scoring  is  done  with  a  portable 
computer  scanner.  We  identify  the  candidates 
who  appear  to  have  high  scores.  If  their  other 
forms  make  them  appear  to  be  eligible,  we  send 
them  pre-employment  papers  to  complete.  All 
candidates’  answer  sheets  and  computer  forms 
are  then  sent  to  an  OPM  facility  in  Macon, 
Georgia,  for  official  scanning  and  entry  into  the 
computer  inventory  of  examinees. 

Candidate  Interview.  The  high  scoring 
candidates  who  appear  to  be  eligible  are  asked  to 
contact  certain  FAA  towers  or  air  route  centers 
near  them  to  schedule  an  interview.  We  have 
trained  a  large  number  of  supervisory  and  man¬ 
agement  level  air  traffic  controllers  throughout 
the  country  to  serve  as  job  interviewers.  The 
training  includes  a  number  of  areas,  including 
EEO. 

The  candidates  complete  the  papers  we  sent 
them  and  bring  them  to  the  interview.  This 
paperwork  helps  us  determine  if  the  candidates 
are  eligible  for  the  job.  Also  at  the  interview, 
applicants  are  given  a  tour  of  the  facility  in  order 
to  help  them  understand  the  work  of  a  controller. 
The  interview  consists  of  both  information  gath¬ 
ering  by  the  interviewer  and  time  spent  explain¬ 
ing  and  showing  the  job  to  the  applicant.  We  find 
this  necessary  because  often  young  people  have 
an  idealized  image  of  the  occupation  and  later 
drop  out  of  training  claiming  they  discovered  "it 


isn’t  for  them."  For  example,  we  have  been 
surprised  by  the  small  number  of  young  people 
who  really  understand  what  we  mean  when  we 
say  that  controllers  work  rotating  ..hifts,  as  well 
as  weekends  and  holidays. 

Medical  Examination.  If  the  interviewer  is 
satisfied  with  the  applicant,  he  or  she  will  sched¬ 
ule  the  person  for  a  medic.u  examination  either 
with  an  FAA  physician  oi  an  aviation  medical 
examiner  under  contract  with  the  FAA. 

We  have  a  strict  set  of  medical  standards  that 
applicants  must  meet.  The  vision  and  hearing 
standards  are  especially  tough.  The  FAA  has 
done  work  to  validate  its  medical  requirements, 
although  some  are  still  controversial. 

A  part  of  the  medical  requirements  has  to  do 
with  emotional  health.  Generally,  Federal  agen¬ 
cies  are  prohibited  from  using  personality  tests  in 
selection.  However,  at  the  interview,  we  ad¬ 
minister  two  forms  of  the  16PF  as  part  of  the 
medical  examination.  It  is  scored  in  the  Office 
of  Aviation  Medicine  in  Washington,  D.  C.  using 
a  "controller  key."  The  test  indicates  who  must 
go  on  for  further  evaluation  by  a  clinical  psy¬ 
chologist  and  psychiatrist.  It  is  not  used  as  an 
absolute  screen-out.  About  2  percent  of  all 
applicants  must  go  on  for  further  evaluation. 

Applicants  also  undergo  urinalysis  for  drug 
use. 

One  reason  that  an  interviewer  may  not  be 
satisfied  with  an  applicant  is  that  the  person  may 
have  at  some  time  taken  illegal  drugs.  We  ask 
candidates  about  that  specifically.  If  they  have 
something  like  drug  use  in  their  background,  we 
will  delay  the  medical  examination  until  comple¬ 
tion  of  the  background  check. 

Background  Investigations.  All  controller 
applicants  are  subject  to  a  thorough  background 
check  before  we  hire  them.  The  FAA  tends  to 
be  wary  of  people  with  driving  while  intoxicated 
convictions,  drug  convictions,  most  criminal 
convictions,  other  than  honorable  discharges  from 
the  military,  and  large  numbers  of  minor  traffic 
violations.  The  background  check  can  be  a  time 
consuming  process.  Much  of  the  work  used  to 
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i.e  done  by  other  agencies.  One  way  the  FAA 
has  expedited  hiring  has  been  to  take  over  the 
routine  portions  of  that  investigation  on  behalf 
of  these  agencies. 

If  the  interviewer  recommends  the  applicant, 
or  at  least  does  not  object  to  the  person,  the 
person’s  medical  exam  indicates  no  problems, 
and  the  background  check  is  clear,  then  the 
person  will  probably  be  hired  if  positions  are 
available. 

The  Screen.  However,  that  is  not  the  end  of 
the  matter  by  any  means.  The  next  step  is  to 
send  the  person  to  the  FAA  Academy  in 
Oklahoma  City,  Oklahoma.  We  hire  the  in¬ 
dividuals,  that  is,  give  them  their  civil  service 
appointments,  when  they  start  the  Academy. 
Because  of  the  vagaries  of  the  federal  budget 
process,  it  is  not  uncommon  for  good  candidates 
to  wait  to  be  given  a  firm  job  offer  in  the  form 
of  a  date  to  report  to  the  Academy.  The  wait  can 
last  several  months  if  there  is  a  backlog  of 
cleared  candidates  waiting  to  report. 

When  the  candidates  finally  do  report,  they 
enter  an  unusual  program  called  the  FAA  Acad¬ 
emy  screening  course.  This  program  was  begun 
in  1976  at  the  direction  of  Congress.  The  pur¬ 
pose  of  the  screening  course  is  to  reduce  attrition 
in  the  field  training  by  identifying  those  can¬ 
didates  with  the  greatest  potential  for  success  in 
the  occupation.  Students  who  do  not  pass  the 
Academy  screening  program  have  their  employ¬ 
ment  terminated. 

The  screening  program  is  a  selection  device. 
It  is  a  second  stage  to  our  selection  system, 
designed  to  eliminate  those  people  unlikely  to 
succeed  in  the  field  training.  The  Screen,  as  we 
call  it,  is  effectively  a  mini-training  and  evalu¬ 
ation  procedure.  We  teach  newly  hired  in¬ 
dividuals  a  simplified  set  of  air  traffic  rules. 
Then,  they  apply  those  rules  in  a  simulated  air 
traffic  control  environment. 

The  OPM  civil  service  test  is  a  fairly  good 
predictor  of  success  in  the  screening  course. 
Even  so,  since  1985,  the  pas.;  tte  at  the  Acad¬ 
emy  has  been  about  60  percent  In  part,  this  is 
because  we  do  no!  always  hav.  the  luxury  of 


hiring  only  those  people  in  the  very  highest  score 
ranges  on  the  test.  For  a  number  of  reasons,  we 
have  not  had  enough  of  those  people  available  in 
the  past. 

The  Screen  has  been  the  subject  of  con¬ 
siderable  criticism  recently.  In  part,  this  is 
because  it  is  easy  to  confuse  this  personnel 
evaluation  procedure  with  training.  There  are 
other  criticisms.  It  is  resource  intensive.  The 
entire  process  takes  about  9  weeks.  During  this 
time,  the  FAA  is  paying  these  persons  a  salary, 
plus  a  certain  amount  for  living  expenses.  It  costs 
the  government  about  $10,000  to  send  someone 
to  the  Academy  screening  course.  Moreover,  we 
hire  people,  then  conduct  a  selection  activity, 
then  terminate  the  employment  of  those  who  do 
not  pass.  Some  criticize  the  content  of  the  Screen 
activities.  For  example,  the  Screen  tasks  are  not 
radar  simulations  and  since  the  Screen  was 
developed,  much  more  ATC  work  is  done  with 
radar.  The  FAA  is  responding  to  these  issues. 

We  have  a  project  in  process  to  develop  a 
three  to  five  day  second  stage  pre-hire  screening 
procedure.  This  project  will  address  the  issue  of 
whether  we  should  be  testing  with  a  more  radar- 
like  simulation.  Most  people  see  computers  as  a 
remedy.  However,  any  computer  simulation 
system  w  ill  have  to  take  into  account  possible  ad¬ 
verse  impact  that  could  occur  due  to  applicants’ 
lack  of  familiarity  with  keyboards  and  CRT’s.  It 
will  also  have  to  take  into  account  the  logistics  of 
fielding  and  securing  computer  testing  systems  to 
screen  several  thousand  people  a  year. 

Developmental  Assignments.  For  those  who 
graduate  from  the  Screen,  there  remain  place¬ 
ment  and  training.  Students  are  hired  by  specific 
FAA  regions  and  sent  to  the  Academy  as  that 
region’s  employees.  If  the  students  pass  the 
Screen,  they  are  placed  in  facilities  within  the 
region  according  to  three  factors:  agency  need, 
how  well  they  did  at  the  Academy,  and  the 
student’s  choice.  Usually,  the  primary  concern 
is:  "Where  do  we  need  people?"  However,  we 
try  to  place  those  who  do  better  at  the  Academy 
into  en  route  centers  and  the  more  difficult  ter¬ 
minals.  Those  who  do  not  do  as  well  we  try  to 
place  in  less  complex  terminals.  Sometimes,  we 
are  able  io  accommodate  student’s  choices  oi 
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facilities  by  giving  them  a  choice  within  a  range 
of  facilities  of  equal  complexity. 

After  the  placement  phase  at  the  Academy, 
the  students  attend  some  further  Academy  train¬ 
ing  to  prepare  them  for  the  type  of  facility  they 
will  be  attending.  Then,  they  move  on  to  their 
first  facility  assignment.  Those  who  go  to  facili¬ 
ties  that  use  radar  will  return  to  the  Academy 
later  for  radar  class.  Developmental  controllers 
receive  the  bulk  of  their  training  in  their  facili¬ 
ties.  This  is  a  combination  of  classroom  and 
on-the-job  training. 

Our  field  training  is  conducted  on  what  we 
call  an  "up  or  out"  basis.  That  means  that  stu¬ 
dents  must  continue  to  progress  in  their  training 
or  have  their  employment  terminated.  We  do  not 
have  permanent  intermediate  positions  short  of 
full  performance  level  controllers.  However,  we 
do  have  some  facilities  with  more  difficult  work 
than  others.  Therefore,  we  have  a  program  that 
allows  students  to  be  given  a  chance  to  succeed 
in  less  complex  facilities  before  terminating  their 
employment.  Not  everyone  gets  that  second 
chance.  It  depends  on  the  availability  of  posi¬ 
tions  and  on  where  the  person  was  in  the  training 
before  he  or  she  failed.  Someone  who  cannot 
succeed  at  the  very  early  training  in  a  complex 
facility  is  unlikely  to  be  able  to  handle  the  work 
in  a  less  complex  facility.  Someone  who  suc¬ 
ceeds  in  the  early  stages  of  training  at  a  complex 
facility  but  who  cannot  handle  radar  work  might 
succeed  in  a  non-radar  facility. 

When  you  consider  that  our  potential  ap¬ 
plicant  pool  is  shrinking,  it  begins  to  make 
economic  sense  to  try  to  make  sure  that  every 
person  who  could  possibly  succeed  does  succeed. 
Studies  that  the  FAA  has  commissioned  of  its 
field  training  indicate  that  the  current  training 
probably  does  not  do  that. 

For  this  reason,  the  FAA  is  redesigning  its 
training  programs. 

Many  large  companies  will  be  facing  this  kind 
of  problem  in  the  future,  if  they  are  not  already. 
When  you  have  large  numbers  of  very  capable 
applicants  available,  you  can  emphasize  selection 
and  de-emphasize  training.  When  the  available 


applicant  pool  is  small,  you  will  still  want  to 
select  judiciously,  but  you  will  find  yourself  with 
a  greater  training  task  if  you  plan  to  fill  your 
vacancies. 

That  is  not  our  only  approach.  We  are  also 
looking  at  alternative  sources  of  applicants  to  try 
to  develop  a  pool  of  those  who  have  training 
already.  We  are  helping  colleges  and  universities 
develop  programs  to  train  people  to  be  air  traffic 
controllers  and  we  are  looking  at  how  we  can 
best  use  those  trained  as  controllers  in  the  mili¬ 
tary. 

While  the  current  controller  test  accom¬ 
modates  those  with  experience  as  air  traffic 
controllers,  it  may  be  that  as  we  develop  these 
alternative  sources  of  candidates  we  will  want  to 
select  them  using  other  procedures. 

In  summary,  the  FAA’s  employment  of 
controllers  is  representative  of  a  cross  section  of 
issues  facing  personnel ists  and  selection  experts. 
We  conduct  large  scale  entry  level  hiring  into  a 
hi-tech  occupation.  We  rely  on  a  shrinking  pool 
of  young  people  for  applicants  so  we  must  recruit 
aggressively.  We  have  to  comply  with  civil  rights 
rules.  We  have  mental  ability  testing,  medical 
standards,  drug  testing,  background  investiga¬ 
tions,  and  age  restrictions.  We  try  to  use  realis¬ 
tic  job  previews.  We  have  to  gauge  the  value  of 
computers  in  our  examining.  We  have  to  judge 
the  proper  mix  of  selection  and  training.  As  we 
seek  alternative  applicant  sources,  we  may  have 
to  develop  alternative  selection  procedures  that 
suit  those  sources. 
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PROCEDURES  FOR  SELECTION  OF  AIR  TRAFFIC  CONTROL  SPECIALISTS 


The  Air  Traffic  Control  Specialist  (ATCS) 
oci  pation  in  the  Federal  Aviation  Administra¬ 
tion  (FAA)  consists  of  three  specialties,  the  en 
route,  terminal,  and  flight  service  station  (FSS) 
options.  En  route  and  terminal  ATCSs  ensure  the 
separation  of  aircraft  traveling  between  airports 
(en  route)  and  approaching  or  departing  from 
airports  (terminal)  through  the  issuance  of  clear¬ 
ances  (instructions  regarding  allowable  altitudes 
and  directions  of  flight)  to  pilots.  FSS  specialists 
are  another  type  of  ATCS  who  provide  such  pilot 
services  as  giving  weather  briefings,  filing  flight 
plans,  and  helping  locate  lost  aircraft.  However, 
because  FSS  specialists  are  not  responsible  for 
ensuring  aircraft  separation,  the  procedures  used 
to  select  them  differ.  This  presentation  will 
address  only  selection  procedures  used  to  hire  en 
route  and  terminal  specialists. 

First  stage  of  selection:  The  OPM  ATCS 
selection  battery.  A  short  history  of  ATCS 
written  selection  batteries:  Early  ATCS  selection 
research  identified  a  number  of  commercial 
aptitude  tests  which  predicted  performance  in 
ATC  training  programs.  In  addition  to  the  com¬ 
mercial  tests,  other  tests  were  developed  (e.g., 
directional  headings,  air  traffic  problems  [ATP]) 
which  contained  items  representing  activities 
performed  on  the  job. 

As  a  result  of  this  research,  a  selection  battery 
consisting  of  tests  of  Arithmetic  Reasoning, 
Spatial  Relations,  Following  Oral  Directions, 
Abstract  Reasoning,  and  Air  Traffic  Problems 
was  implemented  in  1962  as  the  first  Civil 
Service  Commission  (CSC)  selection  battery  for 
ATCSs.  Prior  to  that  time,  the  only  selection 
criterion  was  prior  experience  in  controlling  air 
traffic  (which  would  have  occurred  in  a  military 
environment). 

The  version  of  the  Office  of  Personnel  Man¬ 
agement  (OPM)  selection  battery  currently  in  use 
was  implemented  in  October  1981,  about  two 
months  after  the  ATCS  strike.  This  battery 
consists  of  the  Multiplex  Controller  Aptitude  Test 
(MCAT),  the  Abstract  Reasoning  Test  (retained 
from  the  CSC  battery),  and  the  Occupational 
Knowledge  Test  (OKT). 


MCAT :  The  MCAT  was  developed  to  re¬ 
place  the  written  aptitude  tests  included  in  the 
previous  version  of  the  CSC  selection  battery 
with  a  test  having  higher  predictive  validity. 
MCAT  items  were  developed  to  require  ex¬ 
aminees  to  use  many  of  the  skills  required  for  the 
control  of  air  traffic  in  a  simulated  air  traffic 
setting. 

The  MCAT  provides  a  set  of  air  route  maps 
showing  routes  of  flight  through  a  sector  of 
airspace.  Aircraft  locations  are  indicated  on  each 
air  route  map,  as  are  locations  where  the  routes 
of  flight  intersect.  A  table  accompanies  each  map 
which  includes  other  relevant  information,  such 
as  aircraft  altitudes,  speeds,  and  planned  routes 
of  flight  (see  Figure  1).  Many  MCAT  items 
require  identifying  aircraft  that  may  conflict  with 
other  aircraft.  Other  items  involve  computing 
time-distance  functions,  interpreting  information, 
and  analyzing  spatial  relations.  The  MCAT 
contiins  110  items  and  is  speeded.  Sixty-five 
minutes  are  allowed  for  taking  the  test. 

Harris  (1986)  conducted  a  construct  validity 
study  to  identify  the  underlying  dimensions  of  the 
MCAT  by  correlating  performance  on  it  with 
performance  on  a  set  of  cognitive  marker  tests. 
She  found  high  correlations  between  the  MCAT 
and  tests  of  Integrative  Processes,  General  Rea¬ 
soning,  Spatial  Orientation,  Logical  Reasoning, 
and  Spatial  Scanning.  The  results  of  a  factor 
analysis  suggested  that  MCAT  had  equal  loadings 
on  two  factors:  a  dimension  emphasizing  the 
organization,  definition,  and  manipulation  of  the 
perceptual  field  (.31),  and  a  cognitive  dimension 
emphasizing  verbal  and  nonverbal  reasoning 
(-32). 

ABSR :  The  Abstract  Reasoning  Test  (ABSR) 
was  the  only  test  retained  from  the  original  CSC 
battery.  ABSR  is  a  50-item  paper-and-pencil  test, 
which  assesses  the  ability  to  infer  relationships 
between  symbols.  Both  letter  series  and  figure 
classification  items  are  included  in  the  test  (see 
Figure  2).  Thirty-five  minutes  are  allowed  to  take 
the  ABSR. 
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MULTIPLEX  CONTROLLER  APTITUDE  TEST  (MCAT) 
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TABLE  1:  Applicant  OPM  ATCS  battery  performance 
April  1985  -  September  1985  applicants 
N= 8,826 


Measure 

Mean 

Std.  Dev 

MCAT 

73.0 

16.6 

ABSR 

31.5 

9.3 

OKT 

29.2 

11.6 

TMC 

75.5 

12.5 

Rating 

76.4 

13.3 

OKT:  Before  1981,  extra  points  were 
awarded  to  an  applicant’s  CSC  rating  based  upon 
their  claim  of  prior  job-related  experience.  The 
Occupational  Knowledge  Test  (OKT)  was  devel¬ 
oped  to  provide  a  more  objective  and  reliable 
measure  of  ATCS  job  knowledge.  Items  on  the 
OKT  cover  seven  knowledge  areas  related  to  air 
traffic  control.  The  OKT  contains  80  items  and 
50  minutes  are  allowed  to  take  this  test. 

The  earned  OKT  score  does  not  count  toward 
qualification,  but  provides  additional  points  for 
applicants  who  already  qualified  on  the  basis  of 
their  performance  on  the  other  selection  tests. 

Computing  the  OPM  rating:  To  obtain  an 
OPM  rating,  the  number  of  MCAT  items  an¬ 
swered  correctly  is  weighted  .8  and  the  number 
of  ABSR  items  answered  correctly  is  weighted 
.2.  The  sum  of  the  weighted  scores  is  trans¬ 
muted  so  that  the  result  (the  Transmuted  Com¬ 
posite;  TMC)  is  from  a  distribution  having  a 
mean  of  70  and  maximum  of  100.  If  the  ap¬ 
plicant’s  performance  on  the  aptitude  test  battery 
results  in  a  score  of  at  least  70  (or  75  for  those 
applicants  without  previous  aviation  experience), 
then  extra  credit  points  are  added  to  the  TMC  for 
Veteran’s  Preference  and  for  sufficiently  high 
performance  on  the  OKT.  The  result  is  the  OPM 
Rating  (RAT). 

Table  I  shows  the  means  and  standard  devia¬ 
tions  for  the  component  tests  included  in  the 
OPM  ATCS  battery.  These  data  were  collected 
from  a  group  of  applicants  who  took  the  tests 
between  April  and  September  1985. 


Over  170,000  applicants  have  been  tested 
using  this  procedure  since  1981.  The  FAA  has 
experienced  several  problems  with  this  test. 
First,  the  MCAT  appears  to  be  learnable.  Van- 
Deventer  (1984)  found  that  mean  MCAT  scores 
increased  as  applicants  repeated  the  test.  He 
observed  about  a  7  point  increase  for  those  who 
repeated  the  test  once  and  about  an  11  point 
increase  for  those  who  repeated  the  test  twice. 
Smaller  increases  (no  more  than  2.5  points)  were 
observed  for  those  repeating  the  ABSR  and  OKT 
tests.  However,  the  score  increases  related  to  test 
repetition  were  not  associated  with  corresponding 
improvements  in  Academy  scores.  As  a  result  of 
this  research,  OPM,  in  October  1985,  limited 
testing  repetitions  to  once  every  18  months  for 
those  applicants  who  earned  a  passing  score. 

In  recent  years,  a  number  of  independent 
organizations  began  offering  courses  to  prepare 
candidates  to  take  the  OPM  battery.  An  issue 
now  facing  the  FAA  is  how  to  assess  the  effect 
of  these  courses.  Preliminary  analyses  suggest 
that  those  taking  the  preparation  courses  have 
lower  Academy  pass  rates  than  oLier  candidates 
with  similar  OPM  scores  who  did  net  take 
preparation  courses.  However,  the  identification 
of  those  who  took  the  courses  was  based  on  self- 
report  data;  thus,  additional  studies  mu«-»  he  done 
to  better  understand  the  effects  of  taking  these 
courses. 

Second  stage  of  selection:  The  ATC 
Academy  Screen  program.  A  short  history  of 
the  ATCS  Screen  program:  In  1975,  congres¬ 
sional  hearings  held  by  the  Committee  on  Gov- 


16 


TABLE  2:  Academy  student  performance 
October  1985  -  September  1986  entrants 


Measure 

N 

Mean 

Std.  Dev. 

Block  Average 

902 

93.2 

8.0 

Comprehensive  Phase  Test 

902 

91.1 

6.9 

Laboratory  Average 

848 

66.6 

12.7 

Instructor  Assessment 

848 

75.0 

9.8 

Technical  Assessment 

848 

45.3 

15.7 

Controller  Skills  Test 

847 

76.8 

12.1 

Final  grade 

847 

73.8 

10.0 

ernment  Operations  concluded  that  the  FAA’s 
hasii  To.  selecting  ATCSs  was  inadequate  to 
reasonably  identify  individuals  with  the  potential 
to  complete  training  successfully.  They  also 
determined  that  ATCS  trainee  attrition  from  the 
occupation  (which  typically  occurred  about  two 
to  three  years  into  training)  was  unacceptably 
high. 

The  Congressional  Committee  recommended 
the  development  of  a  standardized,  centralized 
program  designed  to  identify  and  remove  from 
training  those  candidates  who  did  not  demonstrate 
sufficient  aptitude  to  become  ATCSs.  As  a  result 
of  these  recommendations,  the  FAA  developed 
a  program  designed  to  lower  the  costs  of  attrition 
by  improving  the  selection  of  ATCSs.  Screening 
of  ATCS  applicants  began  in  1976.  Originally, 
the  screening  process  included  two  programs, 
one  for  those  selected  to  enter  the  en  route 
option,  and  the  other  for  those  selected  to  enter 
the  terminal  option.  In  1985,  the  two  programs 
were  consolidated  into  a  single  screening  pro¬ 
gram.  In  this  program,  students  are  not  assigned 
to  an  option  or  facility  until  they  complete  the 
Screen.  Scores  earned  in  the  Screen  contribute 
to  the  placement  decision. 

The  nine-week  Academy  Screen  program  is 
designed  to  assess  the  apuiude  of  individuals 
having  no  prior  knowledge  of  the  occupation  by 
having  them  learn  a  set  of  nonradar-based  air 
traffic  control  rules  and  principles,  then  provid¬ 
ing  a  series  of  laboratory  simulation  problems  in 
which  the  student  demonstrates  the  application  of 


those  principles.  Students  complete  the  laboratory 
problems  by  performing  the  duties  of  an  ATCS 
during  standardized,  timed  scenarios  encompas¬ 
sing  the  movement  of  aircraft  through  a  specified 
airspace.  During  the  problem,  another  student 
performs  the  roles  of  the  aircraft  pilots  and  other 
"controllers"  participating  in  the  scenarios. 
Instructors,  former  ATCSs  who  have  been  train¬ 
ed  to  observe  and  rate  student  performance, 
grade  the  performance  of  the  students.  Labora¬ 
tory  grades  are  comprised  of  two  parts,  the  Tech¬ 
nical  Assessment  (based  on  numbers  and  types  of 
errors  made)  and  the  Instructor  Assessment 
(based  on  the  instructor’s  judgment  of  how  well 
the  student  performed  the  problem  as  compared 
with  other  students  the  instructor  had  rated 
previously.) 

Table  2  shows  means  and  standard  deviations 
for  the  individual  graded  components  of  the 
Screen  program.  These  data  were  collected  for 
students  who  took  the  OPM  test  before  the 
change  in  administration  procedures  was  made  in 
October  1985,  and  entered  the  Academy  program 
on  a  competitive  basis  between  October  1985  and 
September  1986.  Numbers  of  observations  differ 
for  the  components  because  some  people  with¬ 
drew  from  the  program  before  its  completion.  It 
is  clear  that  most  candidates  do  well  in  the 
academic  portions  of  the  program,  but  do  not 
perform  as  well  on  the  laboratory  problems. 

Student  performance  on  the  best  five  of  the 
six  graded  laboratory  problems  administered 
comprises  60%  of  the  final  grade,  in  the  current 
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TABLE  3:  Correlations  of  OPM  and  Academy  Performance  Measures  with  Status  in  Field 
Training  for  402  1986  Screen  Graduates  Assigned  to  En  Route  Option 


OPM  Tests 


unadjusted  adjusted 
r  r 


MCAT 

.09 

.24 

Abstract  Reasoning 

.03 

.04 

OKT 

.06 

.04 

Transmuted  Composite 

.08 

.24 

Rating 

.09 

.35 

Academy  Tests 


Block  Average 

.06 

.10 

Comp.  Phase  Test 

.07 

.09 

Lab.  Problem  Average 

.21 

.36 

Inst  Assessment 

.22 

.37 

Tech  Assessment 

.21 

.30 

Controller  Skills  Test 

.16 

.26 

Final  Score 

.24 

.44 

version  of  the  program.  20%  of  the  final  grade 
is  contributed  by  performance  on  academic, 
paper-and-pencil  tests.  Another  20%  of  the  grade 
is  contributed  by  performance  on  the  Controller 
Skills  Test,  which  is  designed  to  duplicate  the 
activities  required  of  the  controller  in  a  paper- 
and-pencil  format.  Figure  3  shows  the  weights 
of  the  individual  components  used  to  compute  the 
final  grade. 

Field  training:  Candidates  who  are  unsuc¬ 
cessful  in  the  Screen  are  usually  removed  from 
the  FAA.  Those  who  pass  (now  called  "devel- 
opmentals”)  are  assigned  to  an  air  traffic  control 
facility.  Facilities  have  differing  levels  of  com¬ 
plexity  (based  on  the  number  and  type  of  aircraft 
that  operate  there)  and  perform  different  types  of 
ai»  uaaic  control  services.  Upon  arrival  at  their 
facility,  developmentals  undergo  training,  con¬ 
ducted  by  the  facility  in  a  pass/fail  mode,  that 
emphasizes  the  procedures  appropriate  for  that 
facility’s  functions  and  designed  specifically  for 
their  airspace.  The  training  program  requires,  on 
the  average,  2.9  years  for  en  route  developmen¬ 
tals  to  complete  and  between  1  and  2.3  years  for 
terminal  developmentals,  depending  on  the  type 


of  facility  to  which  they  are  assigned. 

Information  on  performance  in  field  training 
has  been  collected  by  the  FAA  since  the  strike  in 
1981.  The  data  were  originally  intended  to  be 
used  as  criterion  measures  against  which  to 
compare  developmental  performance  on  the 
selection  procedures.  However,  these  data  have 
also  been  used  to  provide  evidence  for  Equal 
Employment  Opportunity  (EEO)  investigations, 
congressional  inquiries,  and  also  provide  feed¬ 
back  into  management  decision  making.  The  data 
collected  regarding  field  training  include  informa¬ 
tion  on  developmental  training  disposition 
(attained  FPL  level,  still  in  training,  switched 
facilities,  switched  to  a  less  complex  option, 
failed  and  separated  from  the  GS- 2152  (ATCS) 
series,  or  left  the  FAA  for  reasons  unrelated  to 
performance).  Other  data  include  dates  of  occur¬ 
rence,  OTJ  hours,  grades,  and  global  instructor 
ratings  for  specific  training  phases. 

Predictive  validity  of  the  selection  procedures: 
Some  studies  have  been  done  to  assess  the  valid¬ 
ity  of  the  selection  procedures  discussed  here  for 
predicting  performance  in  later  field 
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TABLE  4:  Prediction  of  Field  Training  Status 


Results  of  Multiple  Regression  Analysis 
Predictors  not  Adjusted  for  Restriction  in  Range 


Variable 

R 

R2 

R2 

Change  F 

Beta 

Partial 

Correlation 

OPM  Rating 

.117 

.014 

.014 

5.47 

.06 

.06 

Academy  Score 

.274 

.078 

.064 

16.63 

.26 

.25 

training.  VanDeventer  (1981)  obtained  attrition 
information  and  supervisor  ratings  for  a  sample 
of  successful  Academy  graduates  about  three 
years  after  their  graduation.  He  found  that  the 
correlations  between  the  Academy  composite 
score  and  field  supervisors’  ratings  were  .56 
(adjusted  for  restriction  in  the  range  of  predictor 
scores)  for  those  in  the  en  route  option.  At  the 
time  he  conducted  the  study,  no  CSC  test  scores 
were  available  for  analysis.  Manning,  Della 
Rocco,  and  Bryant  (1989),  found  a  correlation  of 
.46  (adjusted  for  restriction  in  range)  between 
Academy  score  and  both  field  instructor  ratings 
and  field  training  status. 

Table  3  shows  correlations,  both  unadjusted 
and  adjusted  for  restriction  in  the  range  of  pre¬ 
dictor  scores  for  both  OPM  tests  and  Academy 
performance  measures  with  field  training  status 
for  a  group  of  students  who  graduated  from  the 
Academy  in  1986.  These  students  took  the  OPM 
test  before  the  change  in  administration  proce¬ 
dures  was  made  in  October  1985,  and  entered  the 
Academy  between  October  1985  and  September 
1986.  Field  training  status  is  an  ordinal  variable 
with  the  following  categories:  Reached  FPL, 
Still  in  training,  Switched  options,  and  Failed. 
Those  who  separated  from  the  occupation  for 
reasons  unrelated  to  performance  were  excluded 
from  the  analyses. 

It  is  clear  that  the  MCAT  is  the  most  predic¬ 
tive  of  the  OPM  tests.  Interestingly  enough, 
adjusting  for  restriction  in  the  range  of  scores 
reduced  the  correlation  between  the  OKT  and 
field  training  status  because  the  variability  in 
OKT  scores  was  greater  for  the  sample  entering 
the  Academy  than  for  the  applicant  population 
taking  the  test.  This  phenomenon  can  be 


explained  by  differentiated  selection  by  ex¬ 
perience  level.  While  only  5%  of  applicants  who 
earned  no  OKT  points  entered  the  Academy, 
over  20%  of  applicants  who  earned  5  or  more 
OKT  points  were  selected.  The  resulting  sample 
had  greater  variability  of  OKT  scores  than  the 
original  population,  in  which  93%  of  applicants 
earned  no  OKT  points. 

With  regard  to  the  Academy  tests,  scores  on 
laboratory  problems  are  better  predictors  of  field 
training  status  than  are  scores  on  the  academic 
tests.  The  Instructor  Assessment  has  a  slightly 
higher  correlation  with  the  criterion  than  the 
Technical  Assessment,  because  the  range  of  I A 
scores  for  Academy  graduates  was  restricted 
more  than  was  the  range  of  TA  scores.  The 
Controller  Skills  Test,  a  papcr-and-pencil  test 
designed  to  measure  skills  used  in  the  laboratory 
problems,  is  a  better  predictor  of  field  training 
status  than  are  the  more  traditional  paper-and- 
pencil  tests  (Block  Average  and  Comprehensive 
Phase  Test.) 

Two  multiple  regression  analyses  were 
conducted  to  assess  the  relative  contributions  of 
the  OPM  and  Academy  tests  in  predicting  field 
training  status.  The  first  regression  analysis  used 
correlations  of  OPM  and  Academy  tests,  unad¬ 
justed  for  restriction  in  the  range  of  predictors, 
with  field  training  status.  The  results,  shown  in 
Table  4,  suggest  that  both  variables  contribute 
very  little  to  the  prediction  of  ATCS  field  train¬ 
ing  status,  although  Academy  score  accounts  for 
more  of  the  variance  in  field  training  status  than 
does  the  OPM  rating. 

However,  it  must  be  remembered  that  the 
range  of  the  OPM  rating  is  doubly  restricted. 
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TABLE  5:  Prediction  of  Field  Training  Status 


Results  of  Multiple  Regression  Analysis 
Predictions  Adjusted  for  Restriction  in  Range 

R:  Partial 


Variable 

R 

R2 

Change 

F 

Beta 

Correlation 

OPM  Rating 

.353 

.125 

.125 

56.94 

.70 

.29 

Academy  Score 

.511 

.261 

.137 

70.52 

.38 

.39 

Only  applicants  with  scores  higher  than  90  are 
chosen.  Then,  additional  restriction  occurs 
because  of  the  second  stage  of  selection.  Be¬ 
cause  OPM  scores  are  correlated  with  Academy 
scores,  then  restricting  the  range  of  Academy 
scores  by  selecting  only  those  who  score  70  or 
above  further  restricts  the  range  of  OPM  scores 
present  in  the  sample  by  the  time  they  enter  field 
training.  The  range  of  Academy  scores  is  also 
restricted  (but  only  singly)  because  only  those 
who  earn  a  grade  of  70  or  above  are  allowed  to 
progress  into  field  training.  Thus,  it  is  possible 
that  the  low  correlation  of  the  two  predictors 
with  field  training  status  is  an  artifact  of  their 
restricted  variability. 

Another  multiple  regression  analysis  was 
conducted  to  determine  what  would  happen  to  the 
relative  predictability  of  the  OPM  rating  and 
Academy  final  score  if  the  predictor  scores  were 
separately  adjusted  for  restriction  in  range 
(Thorndike,  1949).  The  OPM  score  was  adjusted 
using  rhcrndike’s  3-variable  formula  to  account 
for  the  double  restriction  in  range  due  to  the  two 
stages  of  selection.  Table  5  shows  the  results  of 
the  analysis.  When  adjusted  for  restriction  in 
range,  both  the  OPM  rating  and  Academy  score 
contribute  considerably  more  to  the  prediction  of 
ATCS  field  training  status  than  they  did  in  the 
first  regression  analysis.  By  adjusting  the  OPM 
score  for  double  range  restriction,  it  appears  that 
the  OPM  score  contributes  almost  as  much  to  the 
prediction  of  field  training  status  as  does  the 
Academy  score. 

However,  it  must  be  remembered  that  the 
range  of  the  OPM  rating  is  doubly  restricted. 
Only  applicants  with  scores  higher  than  70  may 


be  selected  to  enter  the  Academy,  and  usually, 
only  those  with  scores  higher  than  90  are  chosen. 

These  results  must  be  interpreted  carefully, 
because  adjustment  for  restriction  in  range  may 
bias  the  resulting  correlations,  especially  if  the 
distribution  of  the  predictor  deviates  significantly 
from  normality.  Using  biased  adjusted  correla¬ 
tions  would  bias  the  results  of  the  regression 
analysis  correspondingly.  While  the  results  of 
this  analysis  may  be  suspect,  they  provide  some 
additional  insight  into  the  relationship  of  the  two 
predictors  with  the  field  training  status  criterion. 

Discussion.  The  Screen  program  receives  a 
certain  amount  of  criticism.  One  problem  is  that 
although  the  Screen  program  is  a  selection  proce¬ 
dure  (because  the  scores  are  used  to  make 
employment  decisions  about  the  candidates) 
several  weeks  of  training  are  provided  initially  to 
prepare  the  student  to  take  the  laboratory  prob¬ 
lems.  Some  training  must  be  provided  because 
it  is  assumed  that  students  have  no  prior  knowl¬ 
edge  of  air  traffic  control.  Thus,  the  program  is 
often  considered,  even  within  FAA,  to  be  a  train¬ 
ing  program  instead  of  a  selection  procedure. 
This  creates  some  confusion  and  dissatisfaction 
within  the  FAA  because  1)  there  is  only  a  60% 
pass  rate  in  a  program  that  many  think  should  be 
training  new  hires  to  be  ATCSs,  and  2)  the 
training  provided  involves  only  some  of  the  job 
tasks  performed  by  most  of  the  controllers  in  the 
field  (because  the  Screen  program  is  based  on 
nonradar  en  route  procedures  while  the  job,  at 
most  en  route  facilities,  involves  radar  control 
procedures).  If  the  program  is  considered  to  be 
a  miniaturized  training-testing  environment  for 
selection,  the  training  that  occurs  should  be 
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TABLE  6:  ATCS  Attrition  by  time  of  entry 


Time  period 

Academy 

Field 

1971-1975 

N/'A 

41% 

1976-1981 

29% 

8% 

1981-1985 

42% 

11% 

viewed  as  related  to  the  testing  effort,  and  not  to 
the  career  development  of  the  controller. 

Using  this  program  has  been  of  value  to  the 
FAA,  because  it  resulted  in  lower  field  attrition 
than  had  been  experienced  before  1976.  Table  6 
compares  attrition  rates  from  the  ATCS  occupa¬ 
tion  for  students  who  entered  the  occupation  at 
three  different  time  periods.  Prior  to  the  im¬ 
plementation  of  the  pass/fail  Screen  program  at 
the  Academy,  about  a  41%  attrition  rate  was 
experienced  in  field  training,  occurring  cn  the 
average  at  about  two  years  into  training  (Henry, 
Ramrass,  Orlansky,  Rowan,  String,  and  Reichen- 
bach,  1975).  By  using  the  Academy  program  to 
screen  employees,  the  majority  of  the  attrition 
was  moved  back  to  occur  during  the  first  three 
months  of  employment,  instead  of  occurring 
several  years  after  hire.  Such  a  system  reduces 
the  burden  on  field  facilities  who  expend  fewer 
resources  training  people  who  will  eventually 
fail,  and  also  reduces  the  burden  on  the  employ¬ 
ees,  who  find  out  sooner  during  their  tenure 
whether  or  not  they  will  continue  to  be  em¬ 
ployed. 

At  the  same  time,  it  is  clear  that  improve¬ 
ments  can  be  made  to  the  methods  used  to  select 
ATCSs.  It  would  certainly  be  less  stressful  for 
an  individual  to  spend  five  days  finding  out 
whether  or  not  he  or  she  has  a  job  than  to  spend 
nine  weeks.  The  time  required  to  train  and  test 
the  candidates  may  be  reduced  if  more  generic 
tasks  are  measured  instead  of  tasks  requiring 
specific  technical  knowledge  to  perform.  The  in¬ 
evitable  problems  associated  with  the  reliability 
of  having  multiple  instructors  observe  and  rate 
the  performance  of  students  may  be  reduced  by 
improving  the  rating  scale  or  the  training  pro¬ 
vided  to  raters  or  by  automating  the  measurement 
and  evaluation  of  performance. 


It  must  also  be  noted  that  field  training  status, 
the  criterion  measure  used  for  the  predictive 
validity  analyses  described  earlier,  is  not  an  ideal 
criterion  for  reasons  often  discussed  in 
Industrial/Organizational  Psychology  texts, 
including  unreliability  and  susceptibility  to  bias. 
It  is  used  primarily  because  the  data  upon  which 
it  is  based  are  easier  to  develop  and  obtain  than 
other  performance  measures.  The  other  training 
performance  criteria  discussed  earlier  are  plagued 
with  the  same  inherent  problems.  At  present,  no 
measures  of  performance  on  the  job  have  been 
developed  for  ATCSs. 

The  FAA  will  soon  sponsor  a  project  to 
address  the  two  problems  described  above.  The 
outcome  of  the  project  will  be  two  products:  a 
more  efficient  second-stage  selection  procedure 
which  will  replace  the  current  Academy  Screen 
program,  and  a  set  of  job  performance  measures 
which  can  be  used  as  criteria  against  which  to 
evaluate  the  validity  of  selection  procedures. 
This  work  is  important,  not  only  to  support  the 
selection  of  today’s  controllers,  but  also  to  pro¬ 
vide  a  basis  for  developing  selection  procedures 
and  criterion  measures  of  performance  for  the 
occupation  as  it  will  begin  to  evolve  over  the 
next  10-20  years. 


References 

Harris,  P.  A.  A  Construct  Validity  Study  of  the 
Federal  Aviation  Administration  Multiplex 
Controller  Aptitude  Test.  U.  S.  Office  of 
Personnel  Management,  December  1986. 

Henry,  J.  H.,  Ramrass,  M.  E.,  Orlansky,  J., 
Rowan,  T.  C.,  String,  J.,  and  Reichenbach, 
R,  E.  Training  of  U  S.  Air  Traffic  Can- 


trollers.  Institute  for  Defense  Analysis 
Report  No.  AD/A-006  603,  January 

1975. 

Manning,  C.  A.,  Della  Rocco,  P.  S.,  and  Bry¬ 
ant,  K.  D.  Prediction  of  Success  in  FAA  Air 
Traffic  Control  Field  Training  as  a  Function 
of  Selection  and  Screening  Test  Performance . 
FAA  Office  of  Aviation  Medicine  Report 
Number  FAA-AM-89-6,  May  1989. 

Thorndike,  R.L.,  Personnel  Selection.  New 
York:  Wiley,  1949. 

VanDeventer,  A.  D.  Field  Training  Perfor¬ 
mance  of  FAA  Academy  Air  Traffic  Control 
Graduates.  Presented  at  the  Annual  Scien¬ 
tific  Meeting  of  the  Aerospace  Medical  As¬ 
sociation,  May,  1981. 

VanDeventer,  A.  D.  A  followup  evaluation  of 
the  new  aptitude  testing  procedures  for  selec¬ 
tion  of  FAA  air  traffic  control  specialists.  In 
VanDeventer,  A.  D.,  Collins,  W.  E.,  Man¬ 
ning,  C.  A.,  Taylor,  D.  K.,  and  Baxter,  N. 
E.  Studies  of  Poststrike  Air  Traffic  Control 
Specialist  Trainees:  I.  Age,  Biographical 
Factors,  and  Selection  Test  Performance 
Related  to  Academy  Training  Success.  FAA 
Office  of  Aviation  Medicine  Report  No. 
FAA-AM-84-6,  June,  1984. 


22 


EVALUATION  ISSUES  IN  THE  SELECTION  OF  AIR  TRAFFIC  CONTROLLERS 


The  major  occupation  of  the  Federal  Aviation 
Administration  (FAA)  is  the  Air  Traffic  Con¬ 
troller  (ATC).  Controllers  perform  a  critical 
safety  function  for  the  FAA:  They  separate 
aircraft,  from  each  other  and  from  other  objects. 
The  strike  by  the  Professional  Air  Traffic  Con¬ 
troller  Organization  (PATCO)  in  1981,  that 
resulted  in  the  firing  of  approximately  11,400  of 
the  17,275  active  controllers  (Sells,  Dailey  & 
Pickrel,  1984),  instigated  increased  public  con¬ 
cern  and  Congressional  inquiries  regarding  airline 
safety.  Deregulation  is  but  one  of  several  factors 
that  continue  to  increase  air  traffic  annually,  that 
makes  air  safety  both  more  difficult  and  more 
important  to  achieve.  Many  of  the  concerns  that 
have  surfaced  about  air  safety  involve  the  recruit¬ 
ment,  hiring,  and  training  of  air  traffic  con¬ 
trollers. 

The  FAA  has  recently  committed  consider¬ 
able  resources  to  developing  a  national  recruit¬ 
ment  plan,  designing  new  selection  procedures, 
and  improving  training  systems.  These  program 
development  efforts  have  been  initiated  in  an 
environment  of  rapid  technological  change.  Air 
traffic  control  equipment  is  becoming  increas¬ 
ingly  automated,  guided  by  the  National  Airspace 
System  (NAS)  Plan.  Work  on  developing, 
implementing,  and  evaluating  human  resource 
management  programs  will  not  occur  in  a 
vacuum,  however.  The  labor  market  has  already 

b.gun  to  change  from  an  abundance  of  qualified 
candidates  to  shortage  conditions  (Johnstone  & 
Packer,  1987). 

The  purposes  of  this  paper  are  to  (a)  describe 
a  systems  approach  for  evaluating  the  effective¬ 
ness  of  any  new  human  resource  management 
(HRM)  program  for  an  occupation,  such  as  a 
selection  system  for  air  traffic  controllers;  (b) 
examine  strategies  for  focusing  the  evaluation  of 
a  single  HRM  program  in  the  broader  context  of 
inter-dependent  HRM  programs;  and  (c)  explore 
trade-offs  between  technical,  operational,  and 
policy/management  decisions  in  a  specific  HRM 
program  area,  such  as  selection. 

A  Systems  Approach  for  Evaluating  Selec¬ 
tion.  In  a  formal  evaluation  (e.g..  Cronbach, 


1982;  Rossi  &  Freeman,  1989)  of  an  HRM 
program,  such  as  a  selection  system,  the  review 
plan  should  consider  the  follow  ng: 

1.  The  overall  goals  of  the  organization; 

2.  The  development  of  an  inter -dependent, 
operational  system  involved  in  recruiting, 
selecting,  training,  and  appropriately  placing 
c^  ....oilers; 

3.  The  effective  monitoring  and  execution  of  a 
comprehensive  research  and  development 
program  that  will  serve  as  the  vehicle  for 
creating  a  new  selection  system; 

4.  The  assessment  of  individual  applicants  for 
selection; 

5.  The  evaluation  of  the  relative  costs  and 
benefits  of  different  components  of  the  HRM 
system. 

Figure  1  depicts  the  FAA’s  human  resource 
management  system  for  air  traffic  controllers. 
This  system,  composed  of  separate  initiatives  for 
planning/automation,  recruitment,  selection 
(technical  and  managerial),  training,  and  or¬ 
ganizational  interventions  highlights  the  need  to 
examine  the  interrelationships  among  all  com¬ 
ponents  in  the  broader  system.  Let  us  consider 
each  of  these  in  turn. 

1.  The  overall  goals  of  the  organization. 

A  recent  summary  of  FAA  goals  includes  the 
following: 

a.  Enhance  U.S.  aviation  safety; 

b.  Build  FAA  culture; 

c.  Develop,  refine,  and  implement  FAA 
programs; 

d.  Assure  effective  and  efficient  use  of 
resources; 

e.  Foster  equal  employment  opportunity 
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(EEO)  programs; 

f.  Reduce  regulatory  and  administrative 
burden; 

g.  Promote  USA/FAA  preeminence  in 
aviation. 

Every  organization  has  goals  and  values,  and 
in  the  FAA  these  are  translated  into  mission 
requirements.  Describing  goals,  and  prioritizing 
them  and  changing  them,  are  not  easy  tasks. 
However,  goals  are  very  important  in  under¬ 
standing  how  an  organization  or  agency  works, 
and  the  HRM  systems  must  reflect  these  goals. 
For  example,  the  first  goal  of  FAA  is  aviation 
safety.  Controllers  and  other  FAA  personnel  are 
in  periodic  tension  with  employees  of  other 
organizations,  such  as  commercial  airlines,  over 
the  relative  riskiness  of  a  specific  set  of  aircraft 
procedures  versus  flight  delays.  Legally,  FAA  is 
the  final  arbiter  of  such  choices,  although  the 
agency  has  been  criticized  both  for  being  too 
strict  and  for  being  too  lenient  with  the  industry 
we  are  required  to  regulate.  The  selection  pro¬ 
cess  for  controllers  has  to  be  able  to  find  and 
train  individuals  who  will,  above  all,  foster 
aviation  safety,  and  also  control  and  monitor  air 
traffic  effectively. 

While  the  unique  requirements  of  a  Federal 
regulatory  agency  are  not  easily  replicable  in  the 
private  sector,  the  aviation  industry  also  has 
important  goals  and  values,  some  of  which  may 
conflict  with  each  other  or  with  the  regulatory 
mission  of  the  FAA.  The  objectives  of  the  HRM 
systems  at  FAA  include  the  NAS  Plan,  FAA 
training,  building  ties  between  government, 
academia,  and  industry,  and  managing  human 
resources.  These  are  some  of  the  ways  FAA  has 
chosen  to  meet  its  goals. 

2.  The  development  of  an  inter-dependent, 
operational  system  involved  in  recruiting, 
selecting,  training,  and  appropriately  placing 
controllers. 

Perhaps  the  most  important  point  about  this 
component  is  the  realization  that  parts  of  the 
personnel  system  that  we  might  have  historically 
considered  independently,  are  not.  An  earliei 


speaker  noted  a  shift  in  the  relative  emphasis 
from  selection  to  training  as  the  character  of  the 
applicant  pool  for  hi-tech  jobs  has  been  changing. 
There  are  many  other  potential  trade-offs  to 
consider.  Let  us  consider  the  components  in 
somewhat  more  detail. 

Automation.  Automation  is  an  important 
issue  at  the  FAA  and  an  expensive  one.  Recent 
and  projected  increases  in  air  travel  both  here 
and  internationally  have  created  the  need  for 
procurements  for  new  computer  and  radar  sys¬ 
tems.  At  FAA,  we  have  instituted  the  NAS  HRM 
Plan  so  that  the  new  equipment  can  be  assimi¬ 
lated  into  our  on-going  systems,  with  minimal 
disruption  in  the  service  we  provide  to  the  air 
traveler.  In  designing  this  equipment,  however, 
who  are  we  designing  it  for?  Current  controllers 
do  not  have  to  have  the  same  kind  of  map  of  the 
sky  in  their  heads,  as  did  the  early  controllers 
who  operated  without  radar.  Can  and  should  we 
design  new  equipment  so  that  the  controller  job 
becomes  less  demanding?  Or  less  interesting? 
What  kinds  of  people  can  we  expect  to  have  as 
controllers  in  1995,  in  2000?  How  do  we  select 
a  work  force  now  with  the  appropriate  skill  mix 
and  train  them  to  perform  evolving  jobs  in  the 
year  2000? 

Recruitment.  A  formal  program  of  recruit¬ 
ment  is  relatively  new  at  FAA,  as  it  is  for  many 
organizations.  Until  recently  we  had  many  more 
well-qualified  applicants  than  we  needed,  but  no 
more.  One  operationalization  of  the  FAA  goal  to 
foster  EEO  programs  is  more  active  and  formal 
recruitment  of  women  and  minorities  for  con¬ 
troller  positions,  a  traditionally  white  male 
occupation.  The  FAA  is  considering  a  variety  of 
alternative  recruitment  approaches.  One  incor¬ 
porates  variants  of  the  "realistic  job  preview" 
approach.  For  the  Co-op  Program  (Co-operative 
Education),  the  FAA  (and  other  Federal  agen¬ 
cies)  contracts  with  selected  institutions  of  higher 
education  to  develop  work-study  programs  for 
qualified  undergraduates.  Students  have  jobs  at 
an  FAA  facility  during  their  college  years,  and 
upon  graduation  have  an  experiential  edge  in  the 
selection  process  if  they  choose  to  apply  for  the 
air  traffic  control  occupation.  The  Pre-Develop- 
mental  Program,  on  the  other  hand,  recruits 
prospects  from  employees  at  the  FAA.  or  at 
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other  Federal  government  agencies,  to  a  con¬ 
troller  upward  mobility  program.  Overall,  co-op 
students  have  been  more  successful  than  average 
in  progressing  through  the  controller  career, 
while  the  Pre-Developmental  students  have  been 
about  average. 

Another  FAA  recruitment  approach  is  being 
developed  at  this  time.  A  number  of  colleges  and 
universities  have  developed  curricula  specifically 
tailored  to  the  air  traffic  controller  career.  Many 
include  courses  that  appear  very  similar  in  con¬ 
tent  to  those  the  FAA  offers  at  the  FAA  Acad¬ 
emy  in  Oklahoma  City  or  in  facilities  nation¬ 
wide.  Until  fairly  recently,  graduates  of  such 
programs  were  given  no  special  consideration  in 
applying  for  air  traffic  controller  positions.  In 
1989,  however.  Congress  authorized  the  Mid- 
America  Research  Consortium,  or  MARC,  to 
develop  a  program  that  would  provide  students 
the  training  necessary  for  them  to  enter  at  a  step 
higher  that  the  normal  procedure.  They  are  to  be 
placed  in  developmental  air  traffic  controller 
positions  at  the  post-FAA  Academy  Screen  level. 
Recently  two  or  three  additional  institutions  have 
signed  formal  agreements  with  FAA  for  similar 
arrangements.  Part  of  the  requirement  assigned 
to  MARC  is  to  develop  performance  measures  so 
that  the  capabilities  of  its  graduates  can  be  ade¬ 
quately  assessed.  In  the  meantime,  they  will  be 
held  to  the  standard  admission  requirements  for 
initial  entry  applicants,  i.e.,  an  acceptable  score 
on  the  OPM  test  battery. 

Some  have  predicted  that  a  possible  outcome 
of  these  collegiate  efforts  could  be  the  reduction 
if  not  elimination  of  preliminary  controller 
training  by  the  FAA.  After  all,  so  the  argument 
goes,  the  Federal  government  accepts  the  gradu¬ 
ates  of  medical  schools  on  face  value;  why  not 
graduates  of  air  traffic  controller  schools?  (The 
military  services  have,  however,  developed  their 
own  medical  school  as  they  were  obtaining 
neither  the  quality  nor  quantity  of  physicians  they 
required.)  There  are  at  least  two  problems  with 
such  proposals  frr  independent  entry,  however: 
The  first  is  the  inadequacy  of  the  current  state  of 
performance  measurement  for  air  traffic  con¬ 
troller,  an  inadequacy  that  will  take  time  and 
effort  to  remediate.  There  is  little  firm  basis  for 
concluding  that  the  graduates  of  Program  X  are 


the  same  as,  if  not  worse  or  better,  than  the 
graduates  from  FAA  programs.  Second,  there  is 
the  policy  of  FAA  accountability.  Air  traffic  con¬ 
trollers  have  legal  responsibility  for  operational 
and  other  errors,  occasionally  for  others  as  well 
as  themselves.  It  may  be  difficult  to  convince 
them  that  universities  and  colleges  provide 
acceptable  training  and  acculturation. 

Selection  -  Initial.  As  a  prior  speaker 
described,  the  FAA  uses  a  four-hour  paper  and 
pencil  battery,  assessing  spatial  and  abstract 
reasoning  as  an  initial  hurdle  for  the  25,000  to 
100,000  candidates  applying  annually  for  con¬ 
troller  jobs.  This  is  the  selection  instrument 
authorized  by  the  Office  of  Personnel  manage¬ 
ment  (OPM)  and  is  called  the  OPM  test.  The 
FAA  has  to  meet  the  same  and  more  EEO  docu¬ 
mentation  and  validation  requirements  as  other 
employers  (U.S.  Equal  Employment  Opportunity 
Commission,  U.S.  Civil  Service  Commission, 
U.S.  Department  of  Labor,  and  U.S.  Depart¬ 
ment  of  Justice,  1978).  We  worry  about  adverse 
impact  and  the  effect  of  more  intense  recruiting 
on  adverse  impact.  We  are  concerned  about 
differential  prediction  and  the  adequacy  of  our 
criterion  measures.  We  have  good  evidence  that 
the  OPM  test  is  a  valid  predictor  of  later  success 
in  the  second  phase  of  selection,  the  FAA  Acad¬ 
emy  Screen.  We  are  feeling  external  and  internal 
pressure  for  criterion  measures  that  are  more 
performance-based  and  that  are  further  along  the 
career  progression  path  (American  Educational 
Research  Association,  American  Psychological 
Association,  and  National  Council  on  Meas¬ 
urement  in  Education,  1985;  Society  for  In¬ 
dustrial  and  Organizational  Psychology,  Int., 
1987). 

There  are  two  specific  concerns  that  may  not 
be  unique  to  the  FAA.  The  first  is  computerized 
testing.  Computers  provide  the  opportunity  for 
greater  job-relatedness  or,  at  least,  greater  face 
validity,  and  their  use  has  been  urged  by  many 
quarters.  However,  the  technical,  logistical,  and 
cost  problems  of  establishing  a  standardized 
assessment  program  equivalent  to  the  current 
decentralized  system  are  formidable.  The  De¬ 
partment  of  Defense,  with  10  times  the  applicant 
pool,  has  been  working  on  such  problems  for 
over  a  decade,  and  they  have  not  yet  imple- 
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merited  a  nationwide  examining  system.  Also, 
the  selection  and  implementation  of  computerized 
assessment  must  be  sensitive  to  the  variable 
levels  of  candidate  experience  and  comfort  with 
computers  and  aviation.  White  males  tend  to 
have  more  of  both.  Fostering  EEO  programs  in 
this  instance  could  have  the  outcome  of  iden¬ 
tification  of  more  women,  minorities,  and  white 
males  with  this  reduced  experience  who  might 
not  otherwise  have  considered,  or  been  con¬ 
sidered  for,  the  controller  job. 

The  second  concern  is  a  corollary  to  the  first, 
and  it  refers  to  test  training  programs.  During 
the  past  year  or  so,  there  has  been  a  proliferation 
of  programs  whose  goal  is  to  help  individuals 
become  air  traffic  controllers,  by  providing  them 
with  information  about  passing  the  OPM  test 
battery.  Some  of  these  programs  are  free;  some 
charge  the  applicant  a  fairly  sizeable  amount  of 
money.  The  test  item  types  used  in  the  OPM  test 
are  among  those  that  are  highly  subject  to  prac¬ 
tice  effects.  We  have  not  yet  completed  a  defini¬ 
tive  study.  Preliminary  evidence  suggests  that 
such  programs  can  be  effective  in  increasing  the 
test  scores  of  at  least  some  applicants  for  con¬ 
troller  jobs  without,  however,  concomitantly 
improving  the  underlying  aptitudes  required  to 
succeed  on  the  job.  The  FAA  is  considering 
further  restrictions  in  an  already  restrictive  policy 
on  retesting.  As  more  systematic  data  accumu¬ 
late,  other  steps  may  be  taken. 

Pre-selection  processing.  The  FAA  has  been 
delegated  direct  hire  authority  for  air  traffic 
controllers  from  OPM,  so  that  only  those  can¬ 
didates  with  sufficiently  high  OPM  test  scores 
(90  and  above,  depending  upon  the  hiring  region) 
will  be  contacted  for  further  consideration.  This 
stage  involves  the  expensive  and  extremely  time 
consuming  processes  of  medical  examinations, 
structured  interviews  at  FAA  facilities,  security 
or  background  investigations,  and  an  opening  in 
the  hiring  quota.  The  FAA  has  been  successful 
in  speeding  up  this  process,  especially  the 
security  checks  for  candidates  who  are  "squeaky 
clean,"  but  this  component  can  take  from  two 
months  to  two  years,  or  more.  This  is  a  large 
amount  of  time  in  a  young  person’s  lifetime,  and 
the  FAA  may  be  losing  a  number  of  well-quali¬ 
fied  candidates  to  the  private  sector,  who  are 


unwilling  or  unable  to  wait  for  a  controller’s  job. 

Selection  -  Academy  Screen.  As  described 
in  other  papers,  the  two-month  long  Screen  at  the 
FAA  Academy  is  a  selection  procedure  of  the 
type  "miniature  training  and  test  situation." 
Students,  termed  "developmental"  controllers,  are 
taught  principles  of  (non-radar)  air  traffic  con¬ 
trol,  and  are  evaluated  on  how  well  they  know 
and  can  apply  these  principles,  via  knowledge 
tests  and  laboratory  problems.  The  passing  rate 
of  Screen  classes  over  the  past  few  years  has 
been  slipping  from  60  percent  to  55  percent  and, 
for  the  past  few  classes,  to  well  under  50  per¬ 
cent.  A  long-term  project  to  replace  the  Academy 
Screen  with  a  week  or  so  of  more  intensive 
assessment  has  begun,  but  will  not  be  ready  for 
implementation  for  several  years.  In  the  mean¬ 
time,  this  component  of  the  air  traffic  controller 
selection  process  appears  to  be  at  best  inefficient. 
It  could  be  that  the  increasing  failure  rate  reflects 
a  decreasing  quality  of  applicants.  Or,  the  current 
Screen  is  insufficiently  adequate  to  the  task  of 
dealing  with  today’s  candidates  for  present  and 
projected  equipment. 

Training  initiatives.  Screen  graduates  are 
currently  placed  in  different  facilities  based 
primarily  on  the  needs  of  the  hiring  region  and 
the  final  Screen  scores,  and  secondarily  on  the 
wishes  of  the  developmentals.  Operational  place¬ 
ment  systems  typically  do  not  have  much  leeway 
for  applicant  choice;  perhaps  the  best  approach 
is  to  be  explicit  about  this  to  the  developmentals. 
Some  advanced  training  for  Screen  graduates  is 
provided  at  the  FAA  Academy,  but  most  is 
obtained  at  the  facilities  where  the  develop¬ 
mentals  are  placed.  A  second  iong  term  FAA 
project  is  to  review  and  revise  the  entire  training 
curriculum.  It  will  be  completed  at  approximately 
the  same  time  as  the  project  for  the  new  Screen. 
The  two  projects  will  dovetail  with  each  other, 
but  it  will  not  be  easy  to  evaluate  the  individual 
contributions  of  each. 

On-the-job  training.  The  FAA  has  Instruc¬ 
tional  Program  Guides  (IPG’s)  listing  the 
accomplishments  each  developmental  controller 
must  demonstrate  before  proceeding  to  the  next 
phase  of  training  and  career  status.  Most  facili¬ 
ties  have  training  managers,  whose  assignment  it 
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is  to  develop  materials  and  programs  to  imple¬ 
ment  the  IPG.  The  goal  of  this  training  is  for  the 
developmental  to  achieve  Full  Performance  Level 
(FPL)  status.  Training  programs  are  not  limited 
to  developmental,  of  course.  There  are  refresher 
and  remedial  programs  and  there  is  training  for 
new  equipment  and  procedures  for  infrequent  yet 
critical  possibilities.  Both  developmental  and  FPL 
controllers  have  to  be  capable  of  learning  what 
they  have  to  with  the  time  and  resources  avail¬ 
able,  while  the  available  training  programs 
should  provide  the  most  effective  means  of  doing 
so  for  the  current  population  of  controllers. 
Evaluation  should  address  (a)  the  standardization 
of  the  on-the-job  training  (OTJ)  across  facilities; 
(b)  the  quality  c '  supervision  received  by  devel¬ 
opmental;  and  (c)  the  effectiveness  of  OTJ 
program  efforts  in  developing  FPL  controllers. 


Retention.  Retention  is  a  critical  issue  in  air 
traffic  control,  or  any  other  occupation,  because 
the  more  controllers  you  retain,  the  fewer  you 
have  to  recruit  and  hire.  Predictable  attrition, 
possible  when  most  of  it  is  retirement,  allows  for 
control  of  applicant  flow.  It  is  far  more  efficient 
to  produce  a  steady  stream  of  new  employees 
than  the  peaks  and  valleys  that  are  common  when 
the  organization’s  recruiting  budget  can  be  chan¬ 
ged  midstream.  A  most  important  concern  for 
controllers  is  pay.  The  FAA  falls  under  the 
general  schedule  pay  scale  applying  to  the 
majority  of  federal  employees,  and  unconnected 
to  differential  living  costs  across  the  United 
States.  Facilities  near  the  Pacific  coast  of  Cali¬ 
fornia,  in  or  around  New  York  City,  for 
example,  face  difficult  problems  in  recruiting 
replacements  because  of  housing  costs.  Fre¬ 
quently,  individuals  wishing  to  transfer  from  such 
facilities  are  prohibited  from  doing  so  until  a 
replacement  can  be  expected,  which  may  take 
months  if  not  years.  The  FAA  currently  has  a 
Pay  Demonstration  project,  in  which  some  of 
these  impacted  facilities  are  permitted  to  provide 
a  20  percent  retention  allowance. 


Other  components  of  retention  include  job 
satisfaction,  employee  assistance  programs, 
unions,  and  performance  incentives.  The 
controller’s  performance,  linked  as  it  is  to  auto¬ 
mation  equipment,  radar,  and  telephones,  is 
monitored  continuously.  Periodic  performance 


evaluations  are  required,  with  mandatory  reviews 
and  possible  retraining  whenever  operations 
errors  or  other  serious  mistakes  occur.  Further, 
the  performance  of  the  controller  work  force,  in 
conjunction  with  other  FAA  safety-related  work 
forces,  impacts  on  broad  aviation  system  factors 
such  as  (a)  safety;  (b)  capacity;  (c)  efficiency; 
and  (d)  security.  It  is  also  legal  for  controllers  to 
have  randomly  announced  drug  tests. 

In  summary,  there  are  many  components  of 
the  air  traffic  controller  HRM  system  at  the 
FAA.  ‘'"'lecting  people  to  become  controllers  is 
influenced  by  all  of  them,  and  influences  all  of 
them,  in  turn.  Evaluating  any  one  component  of 
the  system,  such  as  selection,  requires  considera¬ 
tion  of  all  components. 

3.  The  effective  monitoring  and  execution 
of  a  comprehensive  research  and  development 
program  that  will  serve  as  the  vehicle  for 
creating  a  new  selection  system. 

Even  before  the  passing  of  the  1964  Civil 
Rights  Act  there  were  requirements  for  documen¬ 
tation  of  the  usefulness  of  civil  service  examina¬ 
tions.  Most  jurisdictions,  including  the  Federal 
government,  had  a  few  or  many  jobs  that  were 
included  in  a  merit  system  of  employment.  Merit 
hiring  means  hiring  the  best  qualified  person, 
rank-ordering  candidates  using  their  scores  on  an 
evaluation  procedure  directly  linked  to  perfor¬ 
mance  on  the  job  in  question.  A  traditional 
predictive  test  validation  approach,  including  an 
examination  of  the  psychometric  characteristics  of 
predictors  and  performance  criteria,  and  the 
relationships  between  predictors  and  one  or 
multiple  measures  of  performance  can  be  used 
to  identify  the  measures  that  are  most  predictive 
of  success  at  subsequent  stages  of  career  progres¬ 
sion  of  an  air  traffic  controller.  The  level  of 
detail,  the  characteristics  of  acceptable  evidence 
have,  of  course,  changed  greatly  over  the  past 
three  decades,  but  the  basic  requirement  has  been 
around  a  long  time. 

To  some  eyes  the  state  of  predictors  has 
changed  little  in  the  past  50  years,  which  to 
others  of  us  indicates  that  those  eyes  haven’t  kept 
up  with  the  literature.  For  selection  we  will 
consider  abilities  in  the  cognitive  domain  such  as 
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mental  abilities,  and  perceptual  and  psychomotor 
skills.  We  are  also  considering  non-cognitive 
measures  of  personality,  interests,  and  other 
aspects  of  personal  history.  For  some  occupa¬ 
tions  we  look  at  physiological  indices  as  well, 
such  as  age,  color  blindness,  heart  rate.  A  vari¬ 
ety  of  procedures  will  be  used  for  assessment, 
from  low-tech  paper-and-pencil  to  computers  and 
simulators,  frequently  finding  that  the  paper-and- 
pencil  measure  is  the  most  effective  in  many  (not 
all)  situations.  The  content  and  procedures  of  the 
measures  may  be  more  or  less  similar  to  the 
content  of  the  job.  Finally,  theoretical  advances 
such  as  item  response  theory,  constructed  re¬ 
sponses,  validity  generalization,  generalizability 
theory,  continually  expand  our  horizons,  deepen 
our  understanding,  and  require  review  and 
rewriting  of  our  documentation. 

Before,  we  might  choose  a  selection  device 
based  on  a  fairly  general  job  analysis  combined 
with  a  criterion-related  validity  study  using 
supervisor  ratings  as  criteria,  perhaps  with  a 
sample  size  as  large  as  75  or  100.  Those  days 
are  gone.  While  each  individual  personnel  project 
will  probably  always  require  its  own  job  analysis, 
it  has  become  critical  to  coordinate  the  informa¬ 
tion  obtained  from  successive  job  analyses  of  the 
same  occupation  for  subsequent  projects  to  build 
upon.  The  FAA  is  completing  a  multi-volume  job 
task  analysis  of  air  traffic  controller  occupations, 
based  on  type  of  facility  and  character  of  equip¬ 
ment.  Building  a  selection  system  directly  on  a 
job  task  analysis  is  difficult,  but  building  a 
selection  job  analysis  on  a  prior  job  task  analysis 
makes  a  lot  of  sense. 

While  we  still  do  criterion-related  validity 
studies,  we  try  to  have  much  larger  sample  sizes, 
as  we  know  all  too  well  the  hazards  of  small 
samples.  And,  while  supervisor  ratings  remain 
the  most  popular  criteria  in  published  validation 
research,  we  have  become  increasingly  uncom¬ 
fortable  with  them.  Ratings  are  loaded  with 
components  we  aren’t  interested  in  and  don’t 
want  to  measure.  In  addition  they  can  be  biased 
and  certainly  are  incomplete.  Training  measures, 
typically  predicted  with  higher  validity  than 
supervisor  ratings,  are  somewhat  suspect,  be¬ 
cause  we  aren’t  sure  how  accurate  a  reflection 
the  training  is  of  the  job.  Some  would  prefer  to 


keep  training  measures  only  as  predictors  of  later 
performance.  Measures  of  actual  performance, 
either  from  work  samples  or  simulations,  or  from 
incidentally  maintained  records,  became  more 
desirable  even  as  they  were  acknowledged  to  be 
more  expensive  to  develop  and  to  obtain.  During 
the  1980’s,  the  Department  of  Defense  spon¬ 
sored  a  comprehensive,  multi-service  project  to 
develop  job  performance  measures,  with  the  hope 
of  linking  these  measures  to  selection  standards. 
A  Committee  of  the  National  Academy  of  Sci¬ 
ences  provided  oversight  of  this  project,  and  their 
final  report  is  due  the  end  of  this  year.  The 
major  finding  of  this  project  is  that  reliable  and 
representative  job  performance  measures  can  be 
developed  which  can  be  predicted  by  entrance 
examinations.  Argument  remains  over  whether 
such  performance  measures  are  the  performance 
of  interest,  or  whether  such  performance  meas¬ 
ures  are  but  components  of  a  larger  performance 
space. 

Since  the  equipment  and  hence  required 
behaviors  of  the  air  traffic  controller  job  have 
changed  at  roughly  the  same  frequency  as  the 
civil  rights  case  law,  it  is  necessary  to  continue 
monitoring  and  evaluating  the  entire  selection 
system.  This  includes  continual  review  of  the 
performance  measures  the  selection  system  is 
built  to  predict. 

4.  The  assessment  of  individual  applicants 
for  selection. 

Prior  speakers  have  addressed  some  of  the 
operational  and  logistical  issues  faced  by  the 
FAA  in  assessing  candidates  for  air  traffic 
controller  jobs.  What  will  be  stressed  here  is  the 
obligation  of  Federal  employers  to  evaluate  every 
candidate  fairly  and  in  the  same  way.  Within 
legally  specified  constraints,  such  as  age,  educa¬ 
tion,  and  experience  requirements,  every  Ameri¬ 
can  is  entitled  to  apply  for  any  Federal  govern¬ 
ment  job,  irrespective  of  race,  religion,  gender, 
handicap,  and  ability.  Every  applicant  must  be 
treated  courteously  and  with  respect.  Air  traffic 
control  candidates  frequently  ask  their  Represen¬ 
tative  or  Senator  to  check  the  status  of  their 
applications,  who  in  turn  ask  us.  Members  of 
Congress  cannot  have  a  Federal  agency  provide 
any  special  treatment  for  their  constituents  apply- 
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ing  for  employment  in  a  merit  system.  Occas- 
sionally  we,  or  others  have  erred,  and  we  try  to 
rectify  it.  More  frequently,  the  applicant’s  pre¬ 
dicament  reflects  ignorance  of  which  one  of 
apparently  conflicting  regulations  takes  prece¬ 
dence  in  his  or  her  case. 

Another  restriction  on  public  employers  refers 
to  the  types  of  information  they  may  request 
from  applicants,  as  well  as  how  such  information 
might  be  obtained.  Federal  agencies  are  pro¬ 
hibited  from  asking  direct  questions  about  relig¬ 
ious  practices  and  political  preferences,  topics 
that  might  be  addressed  in  sets  of  biodata  items. 
Offensive  and  invasive  questions,  such  as  appear 
on  some  personality  tests,  can  be  used  only  with 
extensively  documented  justification,  and  under 
restricted  circumstances.  For  example,  controller 
applicants  take  the  16PF  as  part  of  the  medical 
examination.  Only  those  individuals  with  extreme 
scores  are  referred  for  more  intensive  psycho¬ 
logical  examination,  which  may  include  more  of 
the  offensive  and  invasive  items.  Finally,  all 
candidates  for  sensitive  positions  in  the  Federal 
government,  which  includes  air  traffic  con¬ 
trollers,  are  subject  to  security  investigations  that 
include  checking  with  former  employers,  review¬ 
ing  police  records,  and  looking  for  indications  of 
alcohol  and/or  drug  abuse.  Such  investigations 
are  far  more  extensive  and  expensive  than 
administration  of  an  honesty  test.  However,  a 
critical  component  of  all  public  selection  systems 
is  face  validity  which  security  investigations 
have.  It  is  not  just  our  managers  and  profes¬ 
sional  colleagues  who  review  um  instruments  and 
procedures.  Our  reviewers  include  political 
appointees.  Members  of  Congress  and  Senators, 
union  leaders,  and  public  interest  groups  of  all 
kinds.  The  public  employment  system  belongs  to 
the  American  public,  and  each  citizen  has  the 
right  to  question  its  operations. 

5.  The  evaluation  of  the  relative  costs  and 
bent"'ts  of  different  components  of  the  HRM 
svstei  i. 

Since  air  traffic  control  selection  procedures 
will  he  developed  and  fielded  in  the  context  of 
other  HRM  program  initiatives,  the  effect  of 
novel  recruitment  strategies  on  the  composition 
of  the  applicant  pool,  the  role  of  different  entry 


programs  (for  example,  military,  co-ops,  other 
collegiate  programs),  and  the  redesign  of  train¬ 
ing  delivery  systems  will  interact  with  changes  in 
the  selection  procedures.  The  implementation  of 
system-wide  changes  for  the  controller  work 
force  creates  the  need  to  evaluate  the  utility  or 
cost  effectiveness  (Cascio,  1987)  of  recruiting 
strategies,  staffing  requirements,  training  require¬ 
ments,  and  Academy  attrition  rates  against  levels 
of  controller  performance. 

Indeed,  we  can  only  evaluate  our  HRM 
programs  against  levels  of  performance  --  that  is 
the  common  metric.  We  need  to  develop  and 
refine  measures  of  air  traffic  controller  perfor¬ 
mance,  using  work  samples,  simulations,  inter¬ 
views,  ratings,  and  archival  measures.  One  or 
more  sets  of  such  measures  that  are  accepted  as 
representative  of  the  controller  job,  by  its  incum¬ 
bents  and  by  FAA  management,  should  be 
assembled.  There  is  a  need  to  link  levels  on  such 
measures  with  levels  on  predictors,  to  identify 
possible  cut  scores  and  selection  standards.  If  we 
can  do  this,  and  if  we  can  attach  dollar  estimates 
to  the  component  costs  of  our  different  HRM 
programs,  FAA  decision  makers  can  make  better- 
informed  decisions  about  air  safety  and  the  use  of 
FAA  resources  to  achieve  it. 

Need  for  Program  Evaluation.  Since  the 
PATCO  strike,  one  of  the  key  challenges  of  the 
FAA  has  been  to  improve  and  maintain  the 
effective  staffing  levels  of  the  Air  Traffic  Con¬ 
troller  work  force,  particularly  with  respect  to 
staffing  high  traffic  load  facilities  adequately. 
The  FAA  has  initiated  a  number  of  program 
efforts  to  tackle  this  challenge.  Training  systems 
are  being  revised  to  allow  for  specific  simulator 
training  that  will  shorten  the  on-the-job  training 
time  for  new  controllers.  The  amount  of  time  to 
recruit  and  place  applicants  has  been  reduced 
substantially.  Further,  the  planned  research  on 
the  development  of  a  new  ATC  selection  Screen 
will  strengthen  the  agency’s  ability  to  hire  ap¬ 
plicants  who  will  have  a  higher  probability  of 
being  successful  in  training  and  on  the  job.  In 
addition  to  selection  and  training  initiatives,  the 
FAA  has  implemented  pay  demonstration  projects 
to  study  the  feasibility  of  providing  geographi¬ 
cally-based  pay  incentives  and  recruitment  and 
retention  allowances  to  controllers. 
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The  development  and  implementation  of  these 
interdependent  human  resource  management 
programs  create  a  tremendous  need  for  program 
evaluation.  In  a  bureaucracy,  evaluation  can  be 
undertaken  for  a  variety  of  reasons.  Some  of 
these  reasons  are:  (a)  to  make  judgments  about 
program  worth;  (b)  to  increase  the  effectiveness 
of  program  management  and  administration;  (c) 
to  assess  the  usefulness  of  innovative  programs; 
and  (d)  to  satisfy  accountability  demands  and 
answer  stakeholder  (usually  management)  ques¬ 
tions  and  concerns. 

Strategies  for  Conducting  Evaluation.  Al¬ 
though  there  are  a  number  of  approaches  to 
conducting  evaluation,  until  recently  the  FAA 
focused  primarily  on  compliance  types  of  evalu¬ 
ation.  Compliance  evaluations  ask  the  question, 
are  we  doing  things  right?  FAA  policy  requires 
that  comprehensive  evaluations  of  all  designated 
programs  be  conducted  triennially  to  ascertain  the 
effectiveness  and  efficiency  of  program  activities, 
to  verify  internal  compliance  with  established 
procedures  and  practices,  and  to  examine  the 
quality  of  services  and  products  offered  to  FAA 
customers.  In  order  to  implement  existing  FAA 
policy  and  guidance  on  evaluation  activities,  non- 
compliance  types  of  evaluation  need  to  be  asked: 
Are  we  doing  the  right  things? 

Generally,  there  are  five  broad  phases  in 
conducting  an  evaluation  of  any  program.  These 
phases  are  (a)  planning;  (b)  scheduling;  (c) 
execution;  (d)  follow-up  and  tracking;  and  (e) 
trend  analysis.  The  focal  point  of  any  effective 
evaluation  is  the  evaluation  plan.  The  written 
evaluation  plan  should  contain  a  problem  state¬ 
ment  and  background  information.  It  should 
clearly  state  the  evaluation  objectives  and  ques¬ 
tions:  What  is  it  that  management  really  wants 
to  know?  The  audience  for  the  evaluation  find¬ 
ings  should  be  identified  so  that  there  is  upfront 
support  and  involvement  in  the  evaluation  pro¬ 
cess.  Participation  in  the  evaluation  by  manage¬ 
ment  and  other  stakeholders  will  significantly 
increase  the  chances  that  the  evaluation  results 
will  be  used  by  the  organization.  The  evaluation 
plan  should  also  contain  information  on  the 
scope,  methodology,  projected  costs/resources 
needs,  milestones  and  an  outline  of  the  report. 


The  scheduling  phase  of  the  evaluation 
addresses  the  coordination,  logistics,  and 
administration  of  the  program  evaluation.  Para¬ 
mount  in  this  stage  is  assembling  the  correct 
team  to  perform  the  evaluation.  The  FAA,  like 
most  Federal  agencies,  is  just  starting  to  build  a 
formal  internal  staffing  unit  to  conduct  program 
evaluations.  Hence,  the  use  of  an  internal  evalu- 
ation  team,  composed  of  evaluation  staff  and 
other  members  of  the  FAA/HRM  organization, 
will  be  the  vehicle  of  necessity.  Composition  of 
the  team  in  terms  of  skills  and  experience  plays 
a  large  role  in  the  quality  of  the  evaluation 
product.  Further,  it  is  critical  to  establish  effec¬ 
tive  ground  roles  for  the  operation  of  the  team, 
so  that  objectivity  and  fairness  are  maintained 
and  hidden  agendas  do  not  contaminate  the 
execution  of  the  plan  or  interpretation  of  find¬ 
ings. 

The  execution  phase  provides  the  opportunity 
for  the  scientific  method  to  guide  and  temper  fact 
finding  in  an  operational,  applied  work  environ 
ment.  Numerous  evaluation  strategies  can  be 
adopted.  However,  several  common  procedures 
are  consistently  found  in  evaluations  conducted  in 
the  FAA.  The  use  of  available  existing  data  in 
the  form  of  program  records,  statements  of  work, 
directives/orders,  computer  data  files,  and  exter¬ 
nal  evaluation  reports  is  usually  a  starting  point 
in  most  evaluations.  Although  90  percent  of  all 
evaluations  use  existing  data,  the  accuracy  and 
reliability  of  this  information  are  often  difficult  to 
determine.  Another  common  strategy  is  the  use 
of  questionnaires  and  structured  interviews. 
Formal  evaluation  training  of  team  members  is 
particularly  crucial  at  this  stage  to  address  sam¬ 
pling  plans,  control  groups,  response  rates,  pilot¬ 
ing  of  measures,  and  data  analysis  plans.  Another 
important  aspect  of  the  execution  phase  is  the 
review  and  organization  of  the  information 
collected  and  its  interpretation.  The  evaluation 
team  should  reach  consensus  on  the  meaning  of 
information  and  data  gathered  and  any  recom¬ 
mendations  that  are  made.  This  information 
should  be  incorporated  into  stakeholder/manage¬ 
ment  briefings  and  the  formal  report. 

An  action  plan,  based  on  the  recommenda¬ 
tions,  should  be  developed  as  part  of  the  follow¬ 
up  and  tracking  phase  of  the  evaluation  to  insure 
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that  recommendations  are  implemented  in  an 
effective  and  timely  manner.  Then,  applicable 
information  data  bases  should  be  developed  and 
maintained  to  conduct  trend  analyses,  particularly 
in  cases  that  involve  applicant  flow,  training,  and 
performance  data. 

Evaluation  constraints.  A  variety  of  obsta¬ 
cles  can  impede  effective  program  evaluation. 
The  value  management  places  on  evaluation 
activities  determines  whether  program  evaluation 
is  emphasized  within  the  organization  or  indeed 
even  conducted.  Further,  management’s  support 
of  evaluation  drives  the  perception  and  accep¬ 
tability  of  these  activities  in  the  organization  and 
determines  the  level  of  resistance  that  may  be 
encountered.  Time  (short  time  suspense)  and  ade¬ 
quate  resources  (staff  and  money)  are  initial 
hurdles  to  overcome  in  designing  and  planning  an 
evaluation.  The  skill  and  expertise  of  evaluators 
directly  impact  on  the  quality  and  credibility  of 
the  product.  Other  constraints  could  involve 
regulations  that  require  certain  types  of  data 
collections,  access  to  information  that  is  con¬ 
trolled  or  confidential,  program  policies  that 
preclude  making  certain  comparisons  (or  don’t 
allow  control  groups)  or  requirements  for  prior 
approval  by  unions  and  other  groups  to  gather 
information. 

In  the  Federal  bureaucracy,  the  success  of 
evaluation  activities  depends  on  multiple  inter¬ 
dependent  factors:  (a)  a  strong  organizational 
mandate  to  conduct  program  evaluation;  (b) 
managerial  support  (resources)  for  the  function; 
(c)  the  ability  of  evaluators  to  act  independently 
as  internal  consultants  to  the  organization;  (d)  the 
ability  of  evaluators  to  deliver  a  quality  product 
in  a  timely  manner;  (e)  the  development  of 
constructive  recommendations  that  are  translated 
into  action  plans;  and  (f)  effective  follow-up  to 
insure  that  findings  are  used  by  organizational 
decision  makers. 
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DISCUSSION  OF  SELECTION  OF  AIR  TRAFFIC 
CONTROLLERS:  COMPLEXITY,  REQUIREMENTS, 
AND  THE  PUBLIC  INTEREST 


General  Comments.  The  process  of  select¬ 
ing  air  traffic  controller  specialists  (ATCSs),  as 
has  been  discussed  today,  is  obviously  a  task  of 
incredible  complexity.  In  fact,  this  process  is  a 
prototype  for  applying  psychological  theory  and 
research  to  real-world  domains,  given  the  fact 
that  researchers  and  policy-makers  must  attempt 
to  satisfy  sources  with  sometimes  antithetical 
interests  —  government  (e.g.,  FA  A),  union 
(associations  of  ATCSs),  public  policy  (e.g., 
concerns  related  to  age,  sex,  and  race  variables 
in  the  selection  process),  psychological  theory 
(e.g.,  inevitable  conflicts  between  ongoing  devel¬ 
opments  in  developing  new  predictors  for  job 
performance  and  Office  of  Personnel  Manage¬ 
ment  (OPM)  limitations  on  new  tests,  such  as 
those  that  include  a  broad  range  of  non-cognitive 
measures  to  predict  performance),  and  so  on. 
The  speakers  that  preceded  me  today  made  it 
clear  that  excellent  research  and  applications  of 
psychology  to  public  policy  issues  can,  and  are, 
being  carried  out.  What  I  would  like  to  do  in  my 
discussion,  is  to  address  some  of  the  issues  raised 
by  the  speakers  today,  and  also  mention  some 
areas  that  I  think  need  to  be  given  attention  by 
the  FAA  to  both  the  basic  research  and  applied 
psychology  communities. 

1.  Focus  on  specific  aspects  of  the  selec¬ 
tion  process.  One  question  that  occurred  to  me 
in  the  context  of  the  paper  by  Dr.  Manning  had 
to  do  with  an  examination  of  the  full  matrix 
comprised  of  the  hypothetical  states  of  the  world 
and  the  selection  decisions  that  are  made  regard¬ 
ing  ATCS  applicants.  That  is,  as  is  outlined  in 
Figure  1  (and  is  common  to  both  the  Theory  of 
Signal  Detection,  and  Taylor-Russell  tables),  we, 
can  identify  the  two  hypothetical  states  of  the 
world  (shown  as  failure  or  success  as  on-the-job 
(OTJ)  performance  for  an  ATCS.  We  can  also 
look  at  the  decisions  that  are  made  in  the  selec¬ 
tion  system  ("fail"  or  "success"  in  the  ATCS 
Screen). 

With  this  perspective  in  mind,  we  have  heard 
much  about  the  "wash-out"  rate  in  the  ATC 


Screen.  Such  numbers  describe  the  rate  of  "fail¬ 
ure"  in  the  Screen,  but  as  can  be  seen  from 
Figure  1,  the  numbers  do  not  provide  sufficient 
information  to  understand  the  full  decision  pro¬ 
cess.  For  example,  1  wonder  a  great  deal  about 
what  is  the  distribution  of  false  negatives  in  this 
system  (i.e.,  what  percent  of  Screen  wash-outs 
are  false  negatives)?  These  are  persons  who 
would  ultimately  have  suceeded  as  full-perfor¬ 
mance  level  (FPL)  ATCs,  but  for  the  fact  that 
they  failed  the  Screen. 

The  importance  of  the  information  pertaining  to 
false  positives  and  false  negatives  is  critical  to 
making  a  fully  rational  (or  optimal)  set  of  deci¬ 
sions  about  cut-scores  for  Screen  pass/fail  deter¬ 
minations.  The  overall  utility  of  the  Screen  can 
be  examined  with  such  information,  as  is 
illustrated  in  Figure  2.  Clearly,  positive  utility 
values  are  associated  with  "correct  rejections" 
and  with  "hits"  -just  as  negative  utility  values 
(and  thus  the  solution  of  the  matrix)  can  only  be 
established  with  input  from  a  variety  of  sources, 
one  should  not  ignore  that  such  outcomes  do 
indeed  have  differential  utilities  that  may  be  used 
to  optimize  the  system.  Specifically,  perhaps 
passing  criteria  for  the  Screen  should  be  explic¬ 
itly  predicated  on  the  relative  utilities/disutilities 
of  False  Positives/False  Negatives.  This  is  not 
necessarily  a  recommendation  to  take  pass  more 
applicants  from  the  Screen,  but  to  trade  of  the 
cost  of  OTJ  training  against  the  possible  loss  of 
qualifiable  ATCs. 

2.  General  issues:  What  can  basic  re¬ 
search  contribute  to  this  process?  From  a 
theoretical  perspective,  several  basic  research 
themes  seem  to  have  potential  for  contributing  to 
the  problem  of  predicting  individual  differences 
in  acquisition  and  maintenance  of  ATCS  perfor¬ 
mance.  These  relate  to:  Dynamic  changes  in 
ability  -  skill  relations  during  skill  acquisition 
(e.g.,  Ackerman,  1989,  1990);  Motivation  and 
personality  traits  as  predictors  of  job  perfor¬ 
mance  (e.g.,  see  Kanfer,  1991);  and  Issues  of 
metacognition  and  self-regulation  skills  (Kanfer, 
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Figure  1 .  Screen  and  On-The-Job  Performance  from 
the  perspective  of  signal  detection  theory. 
(O-T-J  =  On-the-job;  C.R.  =  Correct  Rejection) 
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Figure  2.  Screen  and  On-The-Job  Performance 
from  the  perspective  of  utility  analysis. 
(O-T-J  =  On-the-job;  C.R.  =  Correct  Rejection) 
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1 990;  Kanfer  &  Ackerman,  1989).  Each  of  these 
domains  will  be  mentioned  below,  but  the  refer¬ 
ences  cited  here  provide  more  extensive  descrip¬ 
tions  of  the  domains  under  consideration  and 
their  potential  applications. 

3.  Age  issues.  Given  changing  demo¬ 
graphics  (i.e.,  an  increasing  median  age  of  the 
U  S.  population),  maybe  it  is  time  to  see  what 
remediation  there  may  be  for  older  ATCSs  - 
either  in  terms  of  training,  equipment  (hardware, 
software)  or  some  combination  of  both,  rather 
than  depend  on  an  increasingly  limited  population 
of  voting  adults  to  be  selected  and  trained  as 
ATCSs. 

Note  the  history  of  human  factors  (fitting  the 
person  to  the  machine  |pre-World  War  11 1  vs. 
fitting  the  machine  to  the  operator  [post-World 
War  II])  (e  g.,  see  Fitts,  1947;  or  see  Sanders  & 
McCormick,  1987  for  a  review  of  the  history  of 
human  factors).  What  aspects  of  the  older  ATCS 
are  problematic0  (Much  research  is  go:  ;  ,  on  in 
other  domains,  for  example,  for  pilot;,  and  for 
other  demanding  tasks.)  This  is  a  general  issue 
that  strongly  suggests  that  the  selection/training 
program  design  be  integrated  with  hardware/soft¬ 
ware  development  and  with  information  about  a 
potenfr.l  job  applicant  pool  (e.g.,  a  "systems" 
approach). 

However,  just  because  Congress  has  mandated 
age  restrictions  for  the  current  system  does  not 
mean  that  given  a  change  in  the  human-machine 
interface  system,  the  age  restrictions  cannot  be 
relaxed.  New  research  is  being  conducted  in  this 
area  that  examines  skill  acquisition  and  retention 
issues  in  older  adults  (e.g.,  see  Rogers  &  Fisk, 
1990).  The  FA  A  should  consider  initiating 
research  on  the  acquisition  and  maintenance  of 
skills  by  ATCSs,  perhaps  in  concert  with  other 
governmental  agencies  that  focus  on  general 
issues  of  aging  (e.g.,  the  National  Institute  for 
Aging).  The  time  to  start  basic  and  applied 
research  in  this  domain  is  now ,  not  after  a  seri¬ 
ous  shortage  of  qualified  applicants  for  ATCS 
positions  is  encountered. 

4.  What  is  the  role  of  confounding  selec¬ 
tion  and  classification,  that  is,  using  scores  to 
assign  students  to  tasks  of  differing  challenge? 


Drs.  Aul  and  Wing  mentioned  that  assignment  of 
Screen  graduates  is  often  done  in  conjunction 
with  their  performance  scores  on  the  Screen. 
That  is.  high-scoring  developmental  are  often 
assigned  to  the  most  challenging  facilities  across 
the  country,  and  those  with  lower  (but  still 
passing)  scores  are  olten  assigned  to  facilities  that 
have  lower  demands  (e.g..  less  traffic).  One 
might  assume  that  this  process  has  the  impact  of 
reducing  variability  of  OTJ  performance,  and 
thus  attenuating  predictive  validities  of  selection 
measures.  However,  some  attempt  sho  ild  be 
made  to  take  this  inhumation  into  account  stalls 
tically  in  order  to  get  an  unbiased  assessment  ot 
the  validity  of  the  various  tests  anJ  Screen 
scores. 

5.  Test  coaching  issue.  Dr  Manning 
reported  some  information  regarding  the 
"learnability"  of  the  OPM  test.  Many  possibilities 
exist  for  a  study  of  this  phenomenon.  Rather  than 
just  restricting  access  to  repeated  testing,  it  may 
perhaps  be  more  fruitful  to  make  test  familiariza¬ 
tion  nearly  universal  (as  with  the  Scholastic 
Aptitude  Tests  [SAT]  and  the  Graduate  Records 
Examination  [GRE]).  or  design  the  test  so  that 
changes  in  test  performance  reflect  actual 
changes  on  the  "construct"  of  interest,  rather 
than  inappropriateness  of  the  test  Dave  Lohman 
(at  the  University  of  Iowa)  and  I  have  found 
some  results  in  using  test  practice,  coupled  w  ith 
informative  feedback,  in  developing  spatial 
abilities  measures  that  are  of  higher  overall 
validity  for  task  performance,  especially  when 
the  task  is  followed  over  skill  acquisition  trials. 
(For  more  details,  see  Ackerman  &  Lohman. 
1990.)  Ultimately  the  focus  comes  back  to  the 
classic  though  somewhat  controversial  issue  of 
aptitude/ability/achievement  measurement.  What 
is  it  that  the  FAA  is  after  when  it  goes  about 
selecting  applicants  for  ATCS  positions0  Is  it 
some  stable  ability,  or  is  it  a  level  of  perfor¬ 
mance  obtained  from  drill  and  practice,  or  from 
mere  exposure  to  similar  test  items0  When 
examinees  differ  greatly  in  background  or  ex¬ 
perience  with  novel  spatial  figures,  it  may  be 
beneficial  from  both  validity  and  public  policy 
perspectives  (vis-a-vis  sex  and  race  differences 
in  test  scores)  to  provide  opportunities  for  at¬ 
tenuating  these  spurious  influences  tin  test  perfor¬ 
mance  via  provision  of  test  practice  --  if.  as  in 
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the  case  of  our  research,  tests  can  be  designed 
that  increase  in  validity  with  practice. 

6.  What  about  "tracking"  the  training 
program,  in  order  to  take  advantage  of 
aptitude-treatment  interactions  (ATI’s)?  It 
seems  as  if  the  outcome  (i.e.,  ATCS  knowledge 
and  skills)  should  be  the  criterion  for  ATCS 
selection,  not  equal  treatment.  Several  pos¬ 
sibilities  exist  for  capitalizing  on  aptitude- 
treatment  interactions.  If  the  FAA  is  willing  to 
consider  the  possibility  of  tailored  training  (e.g., 
via  intelligent  computer-aided  instruction),  it  may 
be  useful  to  develop  measures  that  select  for 
asymptotic  FPL  performance,  rather  than  meas¬ 
ures  that  are  more  highly  oriented  to  selection  for 
training  performance  (Screen).  This  is  especially 
important  in  light  of  the  recent  ability-skill 
acquisition  literature  (e.g.,  Ackerman,  1987, 
1988,  1989)with  regard  to  the  abilities  that 
determine  OTJ  performance.  In  brief,  this  re¬ 
search  has  reaffirmed  the  importance  of  general 
and  broad  content  (spatial,  verbal,  numerical) 
abilities  in  early  task  performance  (e.g.,  during 
training),  but  also  includes  a  framework  that 
identifies  other  (e.g.,  psychomotor)  abilities  as 
valid  predictors  of  asymptotic,  skilled  perfor¬ 
mance.  In  this  sense,  one  can  look  for  talent  (for 
FPL  performance)  that  is  not  captured  by  stan¬ 
dard  measures  of  spatial  abilities  and  reasoning. 
This  sort  of  framework  delineates  that  one  should 
no  more  select  a  clerk-typist  or  a  football  player 
on  the  basis  of  general  reasoning  or  spatial 
abilities,  if  psychomotor  abilities  are  better 
predictors  of  skilled  performance.  The  same 
general  framework  may  apply  to  many  aspects  of 
the  ATCS  job. 

7.  One  final  item.  One  last  issue  seems 
pertinent  to  today’s  symposium.  That  is,  it 
appears  to  be  especially  worthwhile  for  the  FAA 
to  consider  alternatives  to  experimentation  in  the 
operational  environment.  Given  the  limited 
amount  of  time  and  the  general  restrictions  to 
creating  control  and  experimental  groups  of 
ATCS  developmentals,  the  FAA  might  consider 
funding  applied-research  in  the  Academy  and  in 
industry  that  can  focus  on  the  fundamental  issues 
related  to  selection  and  training  of  ATCSs, 
withor  t  the  concomitant  worries  related  to  detri¬ 
mental  impact  of  non-optimal  procedures  that 


could  operate  in  the  on-line  selection/training 
environment.  By  separating  these  specific  real- 
world  concerns  of  the  Screen  from  the  research 
domain,  it  should  be  possible  to  develop  and  test 
(in  isolation  or  in  combination  with  other  inter¬ 
ventions)  new  selection  and  training  procedures. 
Once  fully  tested  in  the  laboratory,  it  would  be 
possible  to  move  directly  into  the  operational- 
environment  with  such  improved  measures  or 
procedures,  without  causing  disruptions  of  the 
flow  of  ATCS  training. 

Within  the  context  of  research  possibilities, 
several  constructs  appear  especially  worthy  of 
attention.  One  set  of  constructs  includes  many 
abilities  that  can  be  optimally  taped  using 
dynamic  displays  (such  as  dynamic  spatial  rea¬ 
soning),  for  example,  see  the  work  by  Gibson 
and  his  colleagues  (1947),  and  by  Pellegrino, 
Hunt,  and  their  colleagues  (e.g.,  Pellegrino, 
Hunt,  &  Yee,  1989).  Another  set  of  constructs 
involve  non-cognitive  determinants  of  skilled 
performance  (such  as  self-regulatory  skills),  such 
as  are  discussed  by  Kanfer  (1990).  Many  other 
constructs  are  potentially  important  in  providing 
incremental  validity  for  predicting  individual 
differences  in  both  ATCS  Screen  and  FPL  per¬ 
formance  levels.  A  discussion  of  these  issues  is 
beyond  the  scope  of  this  discussion,  but  in¬ 
terested  persons  might  consult  compendiums  of 
recent  research,  such  as  the  edited  volumes  by 
Ackerman,  Sternberg,  &  Glaser  (1989);  and  by 
Kanfer,  Ackerman,  &  Cudeck  (1989). 
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